pg.ddx.io pgsql-hackers@postgresql.org mailing list archive
help / color / mirror / Atom feedFrom: Jelte Fennema-Nio <postgres@jeltef.nl>
To: Andres Freund <andres@anarazel.de>
Cc: Nazir Bilal Yavuz <byavuz81@gmail.com>
Cc: Peter Eisentraut <peter@eisentraut.org>
Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Subject: Re: ci: namespace ccache by PostgreSQL major version
Date: Fri, 17 Jul 2026 18:14:06 +0200
Message-ID: <DK0ZA2KYTYVT.1RKL2W5BLVOF1@jeltef.nl> (raw)
In-Reply-To: <DK0YVTWK4BW0.22LT8AQ6QZHIR@jeltef.nl>
References: <CAN55FZ0tqR6Xz=iVFLc1BBoLOEHU775ARhcGYwggHA3XLA=oQg@mail.gmail.com>
<71837b73-768c-4f58-b799-a4462ef0e102@eisentraut.org>
<xdw4thhkr4tzkarc5whatj7a736jaz3rfvjae3a3imm2qz4bn6@kuivz5er7yzd>
<gch4hgdyxnrlqsndn5qln4frftabwtscul5fvqhjpv34tu6ool@gfblk57l7j6e>
<CAGECzQQ6auMC+Y8sRySYR7dphR-o8PDw4=L5LvvFFPAw-4GOUg@mail.gmail.com>
<CAN55FZ2FNoGevdRKjD4e=EHXFMexnsDCGTYpBwzP0t5ZcR3m2A@mail.gmail.com>
<CAGECzQRED+rmO8n6hmBKifuCAD2rMaOWTaWkYaiC2Oe6-VP+bQ@mail.gmail.com>
<e7s3m4cf5x5zekytuadq64jqswrlfsn2fckxq55k4bk3otsa5i@s6br4pdbbtkr>
<DK0YVTWK4BW0.22LT8AQ6QZHIR@jeltef.nl>
On Fri Jul 17, 2026 at 5:55 PM CEST, Jelte Fennema-Nio wrote:
> After implementing that I think there's another approach possible too:
> Allow users to trigger a cache build workflow *from the master branch*
> that builds the cache for specific release branches (by checking them
> out and building them).
Something like the attached, all done Claude. I'm gonna enjoy my weekend
now though. Feel free to take this over.
Attachments:
[text/x-patch] 0001-POC-Allow-running-for-a-different-branch-from-the-ma.patch (9.5K, ../DK0ZA2KYTYVT.1RKL2W5BLVOF1@jeltef.nl/2-0001-POC-Allow-running-for-a-different-branch-from-the-ma.patch)
download | inline diff:
From d2fb68ef96b89dabaf1e4dcc95f6d1980157d3ea Mon Sep 17 00:00:00 2001
From: Jelte Fennema-Nio <postgres@jeltef.nl>
Date: Fri, 17 Jul 2026 18:08:54 +0200
Subject: [PATCH] POC: Allow running for a different branch from the master
branch
This can be useful to populate the caches.
---
.github/workflows/pg-ci.yml | 81 +++++++++++++++++++++++++++++++++++--
src/tools/ci/README | 28 +++++++++++++
2 files changed, 105 insertions(+), 4 deletions(-)
diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml
index 5bc5292d2a5..efe71ac4d01 100644
--- a/.github/workflows/pg-ci.yml
+++ b/.github/workflows/pg-ci.yml
@@ -15,6 +15,26 @@ on:
# adding the 'pull_request' event, as naively doing so would often lead to
# running CI twice.
+ # Build the ccache for another branch and publish it in the dispatching
+ # branch's cache scope. GitHub Actions only lets a workflow run restore
+ # caches created on its own branch or on the default branch, so plain
+ # pushes of branches based on a stable branch normally can't reuse that
+ # stable branch's compilation results. But caches created on the default
+ # branch are readable from everywhere. Thus, dispatching this workflow on
+ # master with e.g. REL_19_STABLE as the input makes that branch's ccache
+ # available to all branches (see the 'Derive cache base branch' step for
+ # the restore side). The run simply executes all of CI on the checked out
+ # branch, so it doubles as a full test of that branch. Re-dispatch
+ # occasionally to refresh the cache as the branch progresses.
+ workflow_dispatch:
+ inputs:
+ branch:
+ description: >-
+ Branch to check out and build, instead of the branch the workflow
+ was dispatched on (e.g. REL_19_STABLE)
+ type: string
+ required: false
+
# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo
# contents during checkout.
permissions:
@@ -26,8 +46,15 @@ concurrency:
# neither want to wait for prior runs, nor to cancel them, so that each
# separately pushed commit is tested. We achieve that by setting a unique
# concurrency group when on such a branch.
+ #
+ # workflow_dispatch cache builds (see `on:`) are keyed by the branch they
+ # build, so that a re-dispatch for a branch replaces a still-running
+ # build of it, while builds for different branches run concurrently. The
+ # bare branch name cannot collide with the push groups, which use the
+ # full refs/heads/... ref.
group: |
${{github.workflow }}-${{
+ inputs.branch ||
case(github.ref == 'refs/heads/master' ||
(startsWith(github.ref, 'refs/heads/REL_') && endsWith(github.ref, '_STABLE')),
github.run_id,
@@ -139,10 +166,15 @@ env:
# Note that we need to be careful to use a separator that can't be in branch
# names, otherwise e.g. caches for 'master' might be restored on the
# 'master-pending' branch.
+ #
+ # For workflow_dispatch runs with a `branch` input, the key names that
+ # branch rather than the one the workflow was dispatched on: the cache
+ # describes the sources that were built, not the ref the run executed on
+ # (which determines the cache's scope, see `on:` above).
CACHE_PREFIX_DEFAULT: >-
:${{ github.job }}:${{ github.event.repository.default_branch }}:
CACHE_PREFIX_BRANCH: >-
- :${{ github.job }}:${{ github.ref_name }}:
+ :${{ github.job }}:${{ inputs.branch || github.ref_name }}:
CACHE_SUFFIX: >-
${{ github.run_id }}:${{ github.run_attempt }}
@@ -292,6 +324,35 @@ jobs:
uses: actions/checkout@v6
with:
fetch-depth: ${{ env.CLONE_DEPTH }}
+ # Empty for everything but workflow_dispatch runs with a `branch`
+ # input, and empty means: check out the triggering commit.
+ ref: ${{ inputs.branch || '' }}
+
+ # Determine which stable branch, if any, the checked out sources
+ # belong to, by parsing the version from meson.build (a 'devel'
+ # version means master). The ccache restore below uses this as an
+ # additional fallback: a stable branch's ccache can be published in
+ # the globally readable default branch cache scope by dispatching
+ # this workflow, see `on:` above. This is what makes plain pushes of
+ # branches based on a stable branch start out with that branch's
+ # cache.
+ - &derive_cache_base_step
+ name: Derive cache base branch
+ id: cache-base
+ shell: bash
+ run: |
+ version=$(sed -n "s/^ *version: '\([^']*\)'.*/\1/p" meson.build | head -n 1)
+ restore_key=''
+ case "$version" in
+ *devel | '')
+ ;;
+ *)
+ major=${version%%[!0-9]*}
+ restore_key="ccache:${GITHUB_JOB}:REL_${major}_STABLE:"
+ ;;
+ esac
+ echo "version: $version"
+ echo "restore-key=$restore_key" | tee -a "$GITHUB_OUTPUT"
# We restore both the ccache from the default branch (typically master),
# and from the current branch. This will often allow feature branches to
@@ -302,7 +363,11 @@ jobs:
# could be corrupted, but we zero them out anyway.
- &ccache_restore_default_step
name: "ccache: Restore for default branch ${{github.event.repository.default_branch}}"
- if: ${{ env.ON_DEFAULT_BRANCH == 'false' }}
+ # Skip when the branch restore below would restore the same cache
+ # anyway. Comparing the prefixes, rather than checking
+ # ON_DEFAULT_BRANCH, makes a workflow_dispatch cache build on the
+ # default branch bootstrap from the default branch's cache.
+ if: ${{ env.CACHE_PREFIX_BRANCH != env.CACHE_PREFIX_DEFAULT }}
uses: actions/cache/restore@v5
with:
path: ${{ env.CCACHE_DIR }}
@@ -310,13 +375,15 @@ jobs:
restore-keys: ccache${{env.CACHE_PREFIX_DEFAULT}}
- &ccache_restore_branch_step
- name: "ccache: Restore for branch ${{ github.ref_name }}"
+ name: "ccache: Restore for branch ${{ inputs.branch || github.ref_name }}"
id: ccache-restore-branch
uses: actions/cache/restore@v5
with:
path: ${{ env.CCACHE_DIR }}
key: ccache${{env.CACHE_PREFIX_BRANCH}}${{env.CACHE_SUFFIX}}
- restore-keys: ccache${{env.CACHE_PREFIX_BRANCH}}
+ restore-keys: |
+ ccache${{env.CACHE_PREFIX_BRANCH}}
+ ${{ steps.cache-base.outputs.restore-key }}
- &linux_prepare_workspace_step
name: Prepare workspace
@@ -493,6 +560,7 @@ jobs:
- *nix_sysinfo_step
- *checkout_step
+ - *derive_cache_base_step
- *ccache_restore_default_step
- *ccache_restore_branch_step
- *linux_prepare_workspace_step
@@ -555,6 +623,7 @@ jobs:
- *nix_sysinfo_step
- *checkout_step
+ - *derive_cache_base_step
- *ccache_restore_default_step
- *ccache_restore_branch_step
- *linux_prepare_workspace_step
@@ -645,6 +714,7 @@ jobs:
- *nix_sysinfo_step
- *checkout_step
+ - *derive_cache_base_step
- *ccache_restore_default_step
- *ccache_restore_branch_step
- *linux_prepare_workspace_step
@@ -795,6 +865,7 @@ jobs:
path: ${{ env.MACPORTS_CACHE }}
key: ${{ steps.mp-key.outputs.key }}
+ - *derive_cache_base_step
- *ccache_restore_default_step
- *ccache_restore_branch_step
@@ -1116,6 +1187,7 @@ jobs:
shell: cmd
run: mkdir ${{env.PG_REGRESS_SOCK_DIR}}
+ - *derive_cache_base_step
- *ccache_restore_default_step
- *ccache_restore_branch_step
@@ -1174,6 +1246,7 @@ jobs:
steps:
- *nix_sysinfo_step
- *checkout_step
+ - *derive_cache_base_step
- *ccache_restore_default_step
- *ccache_restore_branch_step
diff --git a/src/tools/ci/README b/src/tools/ci/README
index 642e518c296..24ba82ebbde 100644
--- a/src/tools/ci/README
+++ b/src/tools/ci/README
@@ -43,6 +43,34 @@ https://github.com/<username>/<reponame>/settings/variables/actions and
create a new repository variable named PG_CI_ENABLED, with the value 1.
+Building shared caches for stable branches
+==========================================
+
+GitHub Actions only allows a workflow run to restore compilation caches
+(ccache) created on the run's own branch or on the repository's default
+branch. A branch based on e.g. REL_19_STABLE therefore normally starts
+building from scratch (or from the default branch's cache, which mostly
+misses), even when REL_19_STABLE itself was recently built by CI. This
+makes iterating on backpatches unnecessarily slow.
+
+However, caches created by runs on the default branch are readable from
+every branch. To take advantage of that, the CI workflow can be manually
+dispatched on the default branch with the name of another branch as input:
+
+ gh workflow run pg-ci.yml --ref master -f branch=REL_19_STABLE
+
+Such a run checks out and builds the given branch, while storing the
+resulting caches in the default branch's cache scope. CI runs detect which
+stable branch the sources they're testing belong to, based on the version
+in meson.build, and restore that branch's shared cache if it exists. Since
+the run executes the whole CI workflow, it also functions as a full test of
+the given branch.
+
+Occasionally re-dispatching the workflow refreshes the cache as the stable
+branch progresses. Caches unused for 7 days are evicted by GitHub, so
+branches only being backpatched to rarely may need a re-dispatch anyway.
+
+
Viewing CI results in a GitHub repository
=========================================
--
2.54.0
view thread (19+ messages) latest in thread
Message-ID: <DK0ZA2KYTYVT.1RKL2W5BLVOF1@jeltef.nl>
Permalink: ../DK0ZA2KYTYVT.1RKL2W5BLVOF1@jeltef.nl/
Also on: postgresql.org/message-id/DK0ZA2KYTYVT.1RKL2W5BLVOF1@jeltef.nl
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: postgres@jeltef.nl, andres@anarazel.de, byavuz81@gmail.com, peter@eisentraut.org, pgsql-hackers@lists.postgresql.org
Subject: Re: ci: namespace ccache by PostgreSQL major version
In-Reply-To: <DK0ZA2KYTYVT.1RKL2W5BLVOF1@jeltef.nl>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox