agora inbox for [email protected]help / color / mirror / Atom feed
[PATCH v48 7/7] Teach snapshot builder to skip transactions running REPACK (CONCURRENTLY). 157+ messages / 2 participants [nested] [flat]
* [PATCH v48 7/7] Teach snapshot builder to skip transactions running REPACK (CONCURRENTLY). @ 2026-03-17 19:22 Antonin Houska <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Antonin Houska @ 2026-03-17 19:22 UTC (permalink / raw) During logical decoding, we need to know if particular transaction performed catalog changes because catalog information is needed to construct heap tuples. To be sure that we have enough information of each transaction, the logical decoding cannot start before all the already running transactions have completed. The problem with REPACK (CONCURRENTLY) is that it has XID assigned and writes WAL records marked with it. Thus if another backend runs REPACK (CONCURRENTLY) and tries to setup the logical decoding, it has to wait for the completion of all the other transactions involved in REPACK (CONCURRENTLY). However, REPACK (CONCURRENTLY) does not perform any catalog changes relevant to logical decoding, so the other backends executing this command can ignore it. This patch implements it by adding information about transactions executing the command to the xl_running_xacts WAL record, and by teaching the snapshot builder to use the information. --- src/backend/access/rmgrdesc/standbydesc.c | 15 ++++- src/backend/access/transam/xlog.c | 2 + src/backend/commands/repack.c | 16 +++++ src/backend/replication/logical/snapbuild.c | 51 +++++++++----- src/backend/storage/ipc/procarray.c | 75 +++++++++++++++++++-- src/backend/storage/ipc/standby.c | 8 ++- src/include/storage/standby.h | 2 + src/include/storage/standbydefs.h | 2 + 8 files changed, 144 insertions(+), 27 deletions(-) diff --git a/src/backend/access/rmgrdesc/standbydesc.c b/src/backend/access/rmgrdesc/standbydesc.c index 0a291354ae2..58ebecf4927 100644 --- a/src/backend/access/rmgrdesc/standbydesc.c +++ b/src/backend/access/rmgrdesc/standbydesc.c @@ -21,10 +21,11 @@ standby_desc_running_xacts(StringInfo buf, xl_running_xacts *xlrec) { int i; - appendStringInfo(buf, "nextXid %u latestCompletedXid %u oldestRunningXid %u", + appendStringInfo(buf, "nextXid %u latestCompletedXid %u oldestRunningXid %u oldestRunningXidLogical %u", xlrec->nextXid, xlrec->latestCompletedXid, - xlrec->oldestRunningXid); + xlrec->oldestRunningXid, + xlrec->oldestRunningXidLogical); if (xlrec->xcnt > 0) { appendStringInfo(buf, "; %d xacts:", xlrec->xcnt); @@ -41,6 +42,16 @@ standby_desc_running_xacts(StringInfo buf, xl_running_xacts *xlrec) for (i = 0; i < xlrec->subxcnt; i++) appendStringInfo(buf, " %u", xlrec->xids[xlrec->xcnt + i]); } + + if (xlrec->xcnt_repack > 0) + { + TransactionId *xids_repack; + + appendStringInfo(buf, "; %d xacts_repack:", xlrec->xcnt_repack); + xids_repack = xlrec->xids + xlrec->xcnt + xlrec->subxcnt; + for (i = 0; i < xlrec->xcnt_repack; i++) + appendStringInfo(buf, " %u", xids_repack[i]); + } } void diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 2c1c6f88b74..f1cbdee7618 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -5912,6 +5912,7 @@ StartupXLOG(void) * subxids are listed with their parent prepared transactions. */ running.xcnt = nxids; + running.xcnt_repack = 0; running.subxcnt = 0; running.subxid_status = SUBXIDS_IN_SUBTRANS; running.nextXid = XidFromFullTransactionId(checkPoint.nextXid); @@ -8483,6 +8484,7 @@ xlog_redo(XLogReaderState *record) * with their parent prepared transactions. */ running.xcnt = nxids; + running.xcnt_repack = 0; running.subxcnt = 0; running.subxid_status = SUBXIDS_IN_SUBTRANS; running.nextXid = XidFromFullTransactionId(checkPoint.nextXid); diff --git a/src/backend/commands/repack.c b/src/backend/commands/repack.c index 781a471fc1d..29ba49744eb 100644 --- a/src/backend/commands/repack.c +++ b/src/backend/commands/repack.c @@ -988,6 +988,22 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, if (concurrent) { + /* + * Do not let other backends wait for our completion during their + * setup of logical replication. Unlike logical replication publisher, + * we will have XID assigned, so the other backends - whether + * walsenders involved in logical replication or regular backends + * executing also REPACK (CONCURRENTLY) - would have to wait for our + * completion before they can build their initial snapshot. It is o.k. + * for any decoding backend to ignore us because we do not change + * tuple descriptor of any table, and the data changes we write should + * not be decoded by other backends. + */ + LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE); + MyProc->statusFlags |= PROC_IN_CONCURRENT_REPACK; + ProcGlobal->statusFlags[MyProc->pgxactoff] = MyProc->statusFlags; + LWLockRelease(ProcArrayLock); + /* * The worker needs to be member of the locking group we're the leader * of. We ought to become the leader before the worker starts. The diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9cf499ce7c6..f2008c6877c 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -1172,7 +1172,7 @@ SnapBuildProcessRunningXacts(SnapBuild *builder, XLogRecPtr lsn, xl_running_xact * xmin, which looks odd but is correct and actually more efficient, since * we hit fast paths in heapam_visibility.c. */ - builder->xmin = running->oldestRunningXid; + builder->xmin = running->oldestRunningXidLogical; /* Remove transactions we don't need to keep track off anymore */ SnapBuildPurgeOlderTxn(builder); @@ -1188,9 +1188,9 @@ SnapBuildProcessRunningXacts(SnapBuild *builder, XLogRecPtr lsn, xl_running_xact */ xmin = ReorderBufferGetOldestXmin(builder->reorder); if (xmin == InvalidTransactionId) - xmin = running->oldestRunningXid; + xmin = running->oldestRunningXidLogical; elog(DEBUG3, "xmin: %u, xmax: %u, oldest running: %u, oldest xmin: %u", - builder->xmin, builder->xmax, running->oldestRunningXid, xmin); + builder->xmin, builder->xmax, running->oldestRunningXidLogical, xmin); LogicalIncreaseXminForSlot(lsn, xmin); /* @@ -1275,14 +1275,14 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn * have all necessary catalog rows anymore. */ if (TransactionIdIsNormal(builder->initial_xmin_horizon) && - NormalTransactionIdPrecedes(running->oldestRunningXid, + NormalTransactionIdPrecedes(running->oldestRunningXidLogical, builder->initial_xmin_horizon)) { ereport(DEBUG1, errmsg_internal("skipping snapshot at %X/%08X while building logical decoding snapshot, xmin horizon too low", LSN_FORMAT_ARGS(lsn)), errdetail_internal("initial xmin horizon of %u vs the snapshot's %u", - builder->initial_xmin_horizon, running->oldestRunningXid)); + builder->initial_xmin_horizon, running->oldestRunningXidLogical)); SnapBuildWaitSnapshot(running, builder->initial_xmin_horizon); @@ -1299,7 +1299,7 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn * NB: We might have already started to incrementally assemble a snapshot, * so we need to be careful to deal with that. */ - if (running->oldestRunningXid == running->nextXid) + if (running->oldestRunningXidLogical == running->nextXid) { if (!XLogRecPtrIsValid(builder->start_decoding_at) || builder->start_decoding_at <= lsn) @@ -1378,14 +1378,14 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn /* * c) transition from BUILDING_SNAPSHOT to FULL_SNAPSHOT. * - * In BUILDING_SNAPSHOT state, and this xl_running_xacts' oldestRunningXid - * is >= than nextXid from when we switched to BUILDING_SNAPSHOT. This - * means all transactions starting afterwards have enough information to - * be decoded. Switch to FULL_SNAPSHOT. + * In BUILDING_SNAPSHOT state, and this xl_running_xacts' + * oldestRunningXidLogical is >= than nextXid from when we switched to + * BUILDING_SNAPSHOT. This means all transactions starting afterwards + * have enough information to be decoded. Switch to FULL_SNAPSHOT. */ else if (builder->state == SNAPBUILD_BUILDING_SNAPSHOT && TransactionIdPrecedesOrEquals(builder->next_phase_at, - running->oldestRunningXid)) + running->oldestRunningXidLogical)) { builder->state = SNAPBUILD_FULL_SNAPSHOT; builder->next_phase_at = running->nextXid; @@ -1402,14 +1402,15 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn /* * c) transition from FULL_SNAPSHOT to CONSISTENT. * - * In FULL_SNAPSHOT state, and this xl_running_xacts' oldestRunningXid is - * >= than nextXid from when we switched to FULL_SNAPSHOT. This means all - * transactions that are currently in progress have a catalog snapshot, - * and all their changes have been collected. Switch to CONSISTENT. + * In FULL_SNAPSHOT state, and this xl_running_xacts' + * oldestRunningXidLogical is >= than nextXid from when we switched to + * FULL_SNAPSHOT. This means all transactions that are currently in + * progress have a catalog snapshot, and all their changes have been + * collected. Switch to CONSISTENT. */ else if (builder->state == SNAPBUILD_FULL_SNAPSHOT && TransactionIdPrecedesOrEquals(builder->next_phase_at, - running->oldestRunningXid)) + running->oldestRunningXidLogical)) { builder->state = SNAPBUILD_CONSISTENT; builder->next_phase_at = InvalidTransactionId; @@ -1459,6 +1460,24 @@ SnapBuildWaitSnapshot(xl_running_xacts *running, TransactionId cutoff) if (TransactionIdFollows(xid, cutoff)) continue; + /* Do not wait for transactions running REPACK (CONCURRENTLY). */ + if (running->xcnt_repack > 0) + { + TransactionId *xids_repack; + int i; + + xids_repack = running->xids + running->xcnt + running->subxcnt; + + for (i = 0; i < running->xcnt_repack; i++) + { + if (xid == xids_repack[i]) + break; + } + /* Found? */ + if (i < running->xcnt_repack) + continue; + } + XactLockTableWait(xid, NULL, NULL, XLTW_None); } diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c index cc207cb56e3..033ae68b57e 100644 --- a/src/backend/storage/ipc/procarray.c +++ b/src/backend/storage/ipc/procarray.c @@ -2643,15 +2643,25 @@ GetRunningTransactionData(void) RunningTransactions CurrentRunningXacts = &CurrentRunningXactsData; TransactionId latestCompletedXid; TransactionId oldestRunningXid; + TransactionId oldestRunningXidLogical; TransactionId oldestDatabaseRunningXid; TransactionId *xids; int index; - int count; + int count, + count_repack; int subcount; bool suboverflowed; + TransactionId *xids_repack = NULL; + bool logical_decoding_enabled = IsLogicalDecodingEnabled(); Assert(!RecoveryInProgress()); + /* + * TODO Consider a GUC to reserve certain amount of replication slots for + * REPACK (CONCURRENTLY) and using it here. + */ +#define MAX_REPACK_XIDS 16 + /* * Allocating space for maxProcs xids is usually overkill; numProcs would * be sufficient. But it seems better to do the malloc while not holding @@ -2663,11 +2673,14 @@ GetRunningTransactionData(void) */ if (CurrentRunningXacts->xids == NULL) { + /* FIXME probably fails if logical decoding is enable on-the-fly */ + int nrepack = logical_decoding_enabled ? MAX_REPACK_XIDS : 0; + /* * First call */ CurrentRunningXacts->xids = (TransactionId *) - malloc(TOTAL_MAX_CACHED_SUBXIDS * sizeof(TransactionId)); + malloc((TOTAL_MAX_CACHED_SUBXIDS + nrepack) * sizeof(TransactionId)); if (CurrentRunningXacts->xids == NULL) ereport(ERROR, (errcode(ERRCODE_OUT_OF_MEMORY), @@ -2676,7 +2689,10 @@ GetRunningTransactionData(void) xids = CurrentRunningXacts->xids; - count = subcount = 0; + if (logical_decoding_enabled) + xids_repack = palloc_array(TransactionId, MAX_REPACK_XIDS); + + count = subcount = count_repack = 0; suboverflowed = false; /* @@ -2688,7 +2704,7 @@ GetRunningTransactionData(void) latestCompletedXid = XidFromFullTransactionId(TransamVariables->latestCompletedXid); - oldestDatabaseRunningXid = oldestRunningXid = + oldestDatabaseRunningXid = oldestRunningXid = oldestRunningXidLogical = XidFromFullTransactionId(TransamVariables->nextXid); /* @@ -2697,6 +2713,8 @@ GetRunningTransactionData(void) for (index = 0; index < arrayP->numProcs; index++) { TransactionId xid; + int pgprocno; + PGPROC *proc; /* Fetch xid just once - see GetNewTransactionId */ xid = UINT32_ACCESS_ONCE(other_xids[index]); @@ -2716,6 +2734,21 @@ GetRunningTransactionData(void) if (TransactionIdPrecedes(xid, oldestRunningXid)) oldestRunningXid = xid; + if (logical_decoding_enabled && + TransactionIdPrecedes(xid, oldestRunningXidLogical)) + { + /* + * Backends running REPACK concurrently need to be excluded from + * oldestRunningXidLogical, otherwise the snapshot builder cannot + * proceed in building the initial snapshot. + */ + pgprocno = arrayP->pgprocnos[index]; + proc = &allProcs[pgprocno]; + + if ((proc->statusFlags & PROC_IN_CONCURRENT_REPACK) == 0) + oldestRunningXidLogical = xid; + } + /* * Also, update the oldest running xid within the current database. As * fetching pgprocno and PGPROC could cause cache misses, we do cheap @@ -2723,8 +2756,8 @@ GetRunningTransactionData(void) */ if (TransactionIdPrecedes(xid, oldestDatabaseRunningXid)) { - int pgprocno = arrayP->pgprocnos[index]; - PGPROC *proc = &allProcs[pgprocno]; + pgprocno = arrayP->pgprocnos[index]; + proc = &allProcs[pgprocno]; if (proc->databaseId == MyDatabaseId) oldestDatabaseRunningXid = xid; @@ -2742,6 +2775,19 @@ GetRunningTransactionData(void) */ xids[count++] = xid; + + /* + * Collect XIDSs of transactions running REPACK (CONCURRENTLY). + */ + if (logical_decoding_enabled && + count_repack < MAX_REPACK_XIDS) + { + pgprocno = arrayP->pgprocnos[index]; + proc = &allProcs[pgprocno]; + + if (proc->statusFlags & PROC_IN_CONCURRENT_REPACK) + xids_repack[count_repack++] = xid; + } } /* @@ -2782,6 +2828,19 @@ GetRunningTransactionData(void) } } + /* + * Append the XIDs running REPACK (CONCURRENTLY), if any. + * + * XXX Should we sort the array and use bsearch() when using it? + */ + if (count_repack > 0) + { + for (int i = 0; i < count_repack; i++) + xids[count++] = xids_repack[i]; + } + if (xids_repack) + pfree(xids_repack); + /* * It's important *not* to include the limits set by slots here because * snapbuild.c uses oldestRunningXid to manage its xmin horizon. If those @@ -2791,11 +2850,13 @@ GetRunningTransactionData(void) * increases if slots do. */ - CurrentRunningXacts->xcnt = count - subcount; + CurrentRunningXacts->xcnt = count - subcount - count_repack; CurrentRunningXacts->subxcnt = subcount; + CurrentRunningXacts->xcnt_repack = count_repack; CurrentRunningXacts->subxid_status = suboverflowed ? SUBXIDS_IN_SUBTRANS : SUBXIDS_IN_ARRAY; CurrentRunningXacts->nextXid = XidFromFullTransactionId(TransamVariables->nextXid); CurrentRunningXacts->oldestRunningXid = oldestRunningXid; + CurrentRunningXacts->oldestRunningXidLogical = oldestRunningXidLogical; CurrentRunningXacts->oldestDatabaseRunningXid = oldestDatabaseRunningXid; CurrentRunningXacts->latestCompletedXid = latestCompletedXid; diff --git a/src/backend/storage/ipc/standby.c b/src/backend/storage/ipc/standby.c index de9092fdf5b..fd3e4fe31f3 100644 --- a/src/backend/storage/ipc/standby.c +++ b/src/backend/storage/ipc/standby.c @@ -1189,6 +1189,7 @@ standby_redo(XLogReaderState *record) RunningTransactionsData running; running.xcnt = xlrec->xcnt; + running.xcnt_repack = xlrec->xcnt_repack; running.subxcnt = xlrec->subxcnt; running.subxid_status = xlrec->subxid_overflow ? SUBXIDS_MISSING : SUBXIDS_IN_ARRAY; running.nextXid = xlrec->nextXid; @@ -1359,10 +1360,12 @@ LogCurrentRunningXacts(RunningTransactions CurrRunningXacts) XLogRecPtr recptr; xlrec.xcnt = CurrRunningXacts->xcnt; + xlrec.xcnt_repack = CurrRunningXacts->xcnt_repack; xlrec.subxcnt = CurrRunningXacts->subxcnt; xlrec.subxid_overflow = (CurrRunningXacts->subxid_status != SUBXIDS_IN_ARRAY); xlrec.nextXid = CurrRunningXacts->nextXid; xlrec.oldestRunningXid = CurrRunningXacts->oldestRunningXid; + xlrec.oldestRunningXidLogical = CurrRunningXacts->oldestRunningXidLogical; xlrec.latestCompletedXid = CurrRunningXacts->latestCompletedXid; /* Header */ @@ -1371,9 +1374,10 @@ LogCurrentRunningXacts(RunningTransactions CurrRunningXacts) XLogRegisterData(&xlrec, MinSizeOfXactRunningXacts); /* array of TransactionIds */ - if (xlrec.xcnt > 0) + if (xlrec.xcnt + xlrec.xcnt_repack > 0) XLogRegisterData(CurrRunningXacts->xids, - (xlrec.xcnt + xlrec.subxcnt) * sizeof(TransactionId)); + (xlrec.xcnt + xlrec.xcnt_repack + xlrec.subxcnt) * + sizeof(TransactionId)); recptr = XLogInsert(RM_STANDBY_ID, XLOG_RUNNING_XACTS); diff --git a/src/include/storage/standby.h b/src/include/storage/standby.h index 6a314c693cd..fe4dcdc3def 100644 --- a/src/include/storage/standby.h +++ b/src/include/storage/standby.h @@ -127,10 +127,12 @@ typedef enum typedef struct RunningTransactionsData { int xcnt; /* # of xact ids in xids[] */ + int xcnt_repack; /* # of xacts running REPACK (CONCURRENTLY). */ int subxcnt; /* # of subxact ids in xids[] */ subxids_array_status subxid_status; TransactionId nextXid; /* xid from TransamVariables->nextXid */ TransactionId oldestRunningXid; /* *not* oldestXmin */ + TransactionId oldestRunningXidLogical; TransactionId oldestDatabaseRunningXid; /* same as above, but within the * current database */ TransactionId latestCompletedXid; /* so we can set xmax */ diff --git a/src/include/storage/standbydefs.h b/src/include/storage/standbydefs.h index 231d251fd51..edad609fa9a 100644 --- a/src/include/storage/standbydefs.h +++ b/src/include/storage/standbydefs.h @@ -47,10 +47,12 @@ typedef struct xl_standby_locks typedef struct xl_running_xacts { int xcnt; /* # of xact ids in xids[] */ + int xcnt_repack; /* # of xacts running REPACK (CONCURRENTLY) */ int subxcnt; /* # of subxact ids in xids[] */ bool subxid_overflow; /* snapshot overflowed, subxids missing */ TransactionId nextXid; /* xid from TransamVariables->nextXid */ TransactionId oldestRunningXid; /* *not* oldestXmin */ + TransactionId oldestRunningXidLogical; TransactionId latestCompletedXid; /* so we can set xmax */ TransactionId xids[FLEXIBLE_ARRAY_MEMBER]; -- 2.47.3 --qfkt2ktdpcfeypib-- ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
* [PATCH v6a 1/5] Add GitHub Actions workflow for CI @ 2026-05-28 16:31 Nazir Bilal Yavuz <[email protected]> 0 siblings, 0 replies; 157+ messages in thread From: Nazir Bilal Yavuz @ 2026-05-28 16:31 UTC (permalink / raw) Cirrus CI, which the project has used to run CI, is shutting down on June 1, 2026. Replace it with a GitHub Actions. Github Actions is selected because it has unlimited runner time for public repositories. Github Action currently covers: - SanityCheck - Linux (Autoconf) - Linux (Meson, 32- and 64-bit) - macOS (Meson) - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Jelte Fennema-Nio <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/postgresql-ci.yml | 1026 ++++++++++++++++++++++++++ src/tools/ci/ci_macports_packages.sh | 19 +- 2 files changed, 1042 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/postgresql-ci.yml diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml new file mode 100644 index 00000000000..a7ef0bee94d --- /dev/null +++ b/.github/workflows/postgresql-ci.yml @@ -0,0 +1,1026 @@ +# GitHub Actions CI configuration for PostgreSQL + +name: GitHub Actions CI + +on: + push: + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never cancel in-progress runs on master to ensure all commits are tested. + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + CCACHE_MAXSIZE: "250M" + + # check target for the autoconf builds + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Dldap=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dplperl=enabled + -Dplpython=enabled + -Ddocs=enabled + -Dicu=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Debian Trixie container image used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/';. + LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + _LOG_PATHS: &log_paths | + build*/testrun/**/*.log + build*/testrun/**/*.diffs + build*/testrun/**/regress_log_* + build*/meson-logs/*.txt + + +jobs: + # Parse "ci-os-only: ..." from the commit message and expose flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + runs-on: ubuntu-latest + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + steps: + - id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + set -e + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: needs.setup.outputs.sanitycheck == 'true' + runs-on: ubuntu-latest + timeout-minutes: 15 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern. + options: --privileged + env: + BUILD_JOBS: 8 + TEST_JOBS: 8 + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + steps: + # Anchor reused by other jobs further down. GitHub Actions supports + # YAML anchors/aliases but not merge keys, so the alias copies the + # whole step verbatim. The anchor is resolved at YAML parse time, so the + # alias keeps working even if this job is skipped at runtime. + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-sanitycheck-${{ github.ref_name }}- + ccache-sanitycheck- + + - name: Prepare workspace + run: | + whoami + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + + - name: Configure + run: | + su postgres <<-'EOF' + set -e + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + EOF + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + - name: Test + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + meson test ${MTEST_ARGS} --suite setup + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ + cube/regress pg_ctl/001_start_stop + EOF + + - name: Core backtraces + if: failure() + run: | + mkdir -m 770 /tmp/cores + find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; + src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: sanitycheck-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + # Build & test postgres on Linux in three configurations. + # + # Autoconf: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # + # Meson: + # - Test both 64- and 32-bit builds + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered + # coverage. Also, newer Python insists on changing LC_CTYPE away from C, + # prevent that with PYTHONCOERCECLOCALE. + # + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + linux: + name: Linux - ${{ matrix.name }} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: Autoconf + slug: autoconf + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=address + pg_test_pg_combinebackup_mode: '--copy-file-range' + configure: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} \ + CLANG="ccache clang" + build: | + make -s -j${BUILD_JOBS} world-bin + test: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + logs_paths: | + **/*.log + **/*.diffs + **/regress_log_* + + - name: Meson (64-bit) + slug: meson-64 + cc: ccache gcc + cxx: ccache g++ + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + + - name: Meson (32-bit) + slug: meson-32 + cc: ccache gcc -m32 + cxx: ccache g++ -m32 + sanitizer_flags: -fsanitize=alignment,undefined + pg_test_initdb_extra_opts: '-c io_method=io_uring' + configure: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + build: | + ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + ninja -C build -t missingdeps + test: | + PYTHONCOERCECLOCALE=0 LANG=C \ + meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} + logs_paths: *log_paths + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + env: + BUILD_JOBS: 4 + TEST_JOBS: 8 + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 + CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} + LDFLAGS: ${{ matrix.sanitizer_flags }} + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + + PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} + PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- + ccache-linux-${{ matrix.slug }}- + + - name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p "$CCACHE_DIR" + chown -R postgres:postgres "$CCACHE_DIR" + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed on Linux Meson but it doesn't harm to have + # this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + - name: Configure + run: | + su postgres <<EOF + set -e + ${{ matrix.configure }} + EOF + + - name: Build + run: | + su postgres <<EOF + set -e + ${{ matrix.build }} + EOF + + - name: Test world + run: | + su postgres <<EOF + set -e + ulimit -c unlimited + ${{ matrix.test }} + EOF + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} + path: ${{ matrix.logs_paths }} + if-no-files-found: ignore + + + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + BUILD_JOBS: 4 + # Test performance regresses noticeably when using all cores. 8 works OK. + # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de + # Fix: Needs to be re-tested for GitHub Actions. + TEST_JOBS: 8 + + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *checkout_step + + - name: Sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-macos-${{ github.ref_name }}- + ccache-macos- + + - name: Compute MacPorts cache key + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" + echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + + - name: Restore MacPorts cache + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + restore-keys: ${{ steps.mpkey.outputs.restore-key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: Install dependencies (MacPorts) + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + run: | + export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + + - name: Test world + run: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + - name: Core backtraces + if: failure() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: macos-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-vs: + name: Windows - VS - Meson & ninja + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 8 + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + + MESON_FEATURES: >- + -Dcpp_args=/std:c++20 + -Dauto_features=disabled + -Dtap_tests=enabled + -Dldap=enabled + -Dssl=openssl + -Dplperl=enabled + -Dplpython=enabled + TAR: "c:/windows/system32/tar.exe" + + defaults: + run: + shell: cmd + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + - name: Install dependencies + shell: pwsh + run: | + choco install -y --no-progress --limitoutput diffutils winflexbison3 + # meson + ninja aren't preinstalled on windows-2022. Install via pip + python -m pip install --upgrade meson ninja + + # OpenSSL 1.1 via the slproweb installer (pinned to match the + # version used elsewhere in postgres CI). + curl.exe -fsSL -o openssl-setup.exe https://slproweb.com/download/Win64OpenSSL-1_1_1w.exe + Start-Process -Wait -FilePath ./openssl-setup.exe ` + -ArgumentList '/DIR=d:\openssl\1.1\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' + # The slproweb installer puts libcrypto-1_1-x64.dll / libssl-1_1-x64.dll + # in d:\openssl\1.1\bin\ and updates the system PATH. GH Actions + # snapshots PATH at job start though, so the running job won't + # see those DLLs and initdb.exe would crash silently at runtime. + # Push the bin dir onto GITHUB_PATH so it persists for later steps. + Add-Content $env:GITHUB_PATH "d:\openssl\1.1\bin" + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + perl -mIPC::Run -e 1 + + - name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build %MBUILD_TARGET% + ninja -C build -t missingdeps + + - name: Test world + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-vs-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + TEST_JOBS: 4 # higher concurrency causes occasional failures + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_DIR: D:/a/ccache + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - name: Disable Windows Defender + shell: powershell + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: powershell + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed \ + git bison flex make diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkg-config \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + + - name: Setup socket directory + shell: cmd + run: mkdir %PG_REGRESS_SOCK_DIR% + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-mingw-${{ github.ref_name }}- + ccache-mingw- + + - name: Configure + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + -DTAR=${TAR} \ + build + + - name: Build + run: ninja -C build ${MBUILD_TARGET} + + - name: Test world + run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + + # FIX: We need to collect crashlogs but they are not collected. cdb.exe + # is installed on the runner so it needs to be configured. + - name: Upload logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: windows-mingw-logs-${{ github.run_id }} + path: *log_paths + if-no-files-found: ignore + + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.linux_ci_image }} + env: + BUILD_JOBS: 4 + CCACHE_DIR: /tmp/ccache_dir + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + steps: + - *checkout_step + + - name: Restore ccache + uses: actions/cache@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} + restore-keys: | + ccache-compiler-warnings-${{ github.ref_name }}- + ccache-compiler-warnings- + + - name: Sysinfo + run: | + id + uname -a + cat /proc/cmdline + ulimit -a -H && ulimit -a -S + gcc -v + clang -v + env + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + mkdir -p "$CCACHE_DIR" + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: always() + run: | + ./configure \ + --cache gcc.cache \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert off, dtrace off + - name: clang warnings + if: always() + run: | + ./configure \ + --cache clang.cache \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: always() + run: | + ./configure \ + --cache clang.cache \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} \ + CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + - name: mingw warnings (cross compilation) + if: always() + run: | + ./configure \ + --host=x86_64-w64-mingw32ucrt \ + --enable-cassert \ + --without-icu \ + CC="ccache x86_64-w64-mingw32ucrt-gcc" \ + CXX="ccache x86_64-w64-mingw32ucrt-g++" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: always() + run: | + ./configure \ + --cache gcc.cache \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -C doc + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: always() + run: | + ./configure \ + ${LINUX_CONFIGURE_FEATURES} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..18a06f96119 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -20,13 +20,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$mac...-[A-Za-z]*\.pkg\"" | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or github actions" 1>2 exit 1 fi -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0002-gha-Andres-revisions.patch" ^ permalink raw reply [nested|flat] 157+ messages in thread
end of thread, other threads:[~2026-05-28 16:31 UTC | newest] Thread overview: 157+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2026-03-17 19:22 [PATCH v48 7/7] Teach snapshot builder to skip transactions running REPACK (CONCURRENTLY). Antonin Houska <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]> 2026-05-28 16:31 [PATCH v6a 1/5] Add GitHub Actions workflow for CI Nazir Bilal Yavuz <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox