agora inbox for pgsql-hackers@postgresql.org
help / color / mirror / Atom feedci: namespace ccache by PostgreSQL major version
19+ messages / 5 participants
[nested] [flat]
* ci: namespace ccache by PostgreSQL major version
@ 2026-07-03 14:26 Nazir Bilal Yavuz <byavuz81@gmail.com>
0 siblings, 1 reply; 19+ messages in thread
From: Nazir Bilal Yavuz @ 2026-07-03 14:26 UTC (permalink / raw)
To: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>; +Cc: Andres Freund <andres@anarazel.de>
Hi,
When the Postgres major version is updated, CI fails with [1]:
2026-07-02 17:01:01.938 UTC [828] FATAL: incompatible library
"D:/a/postgresql/postgresql/build/tmp_install/usr/local/pgsql/lib/utf8_and_win.dll":
version mismatch
2026-07-02 17:01:01.938 UTC [828] DETAIL: Server is version 20,
library is version 19.
This happens because the GitHub Actions ccache is restored across a
version bump, so objects built against the previous version can be
reused, producing libraries that no longer match the server version.
Here is an attempt to solve this problem by namespacing ccache by
Postgres major version. I added a 'PG_MAJOR_VERSION' variable to the
CI file and used that as a prefix to ccache key. I made this
'PG_MAJOR_VERSION' variable automatically updated by
'version_stamp.pl' script.
Another solution could be reading the version from build files (e.g
meson.build), but then this read needs to be done at each CI run.
[1] https://github.com/postgresql-cfbot/postgresql/actions/runs/28606913807/job/84829900634
--
Regards,
Nazir Bilal Yavuz
Microsoft
Attachments:
[text/x-patch] v1-0001-ci-namespace-ccache-by-PostgreSQL-major-version.patch (3.0K, ../../CAN55FZ0tqR6Xz=iVFLc1BBoLOEHU775ARhcGYwggHA3XLA=oQg@mail.gmail.com/2-v1-0001-ci-namespace-ccache-by-PostgreSQL-major-version.patch)
download | inline diff:
From d79d51ebf5fa30a20d76e2cb812abed5d684f571 Mon Sep 17 00:00:00 2001
From: Nazir Bilal Yavuz <byavuz81@gmail.com>
Date: Fri, 3 Jul 2026 17:12:01 +0300
Subject: [PATCH v1] ci: namespace ccache by PostgreSQL major version
Add a PG_MAJOR_VERSION variable to the CI workflow and have
version_stamp.pl keep it in sync. The ccache cache keys include it, so
bumping the version starts from a fresh cache instead of reusing stale
objects from the previous major version.
Reported-by: Robert Haas <robertmhaas@gmail.com>
---
.github/workflows/pg-ci.yml | 15 +++++++++++----
src/tools/version_stamp.pl | 3 +++
2 files changed, 14 insertions(+), 4 deletions(-)
diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml
index 5bc5292d2a5..d1ef90ab3f6 100644
--- a/.github/workflows/pg-ci.yml
+++ b/.github/workflows/pg-ci.yml
@@ -136,6 +136,13 @@ env:
ON_DEFAULT_BRANCH: ${{github.event.repository.default_branch == github.ref_name }}
+ # Used to namespace the ccache caches, so that a new major version starts
+ # from a fresh cache rather than reusing stale objects built against the
+ # previous major version.
+ #
+ # Don't set manually, it is automatically set by 'version_stamp.pl' script.
+ PG_MAJOR_VERSION: "20"
+
# Note that we need to be careful to use a separator that can't be in branch
# names, otherwise e.g. caches for 'master' might be restored on the
# 'master-pending' branch.
@@ -306,8 +313,8 @@ jobs:
uses: actions/cache/restore@v5
with:
path: ${{ env.CCACHE_DIR }}
- key: ccache${{env.CACHE_PREFIX_DEFAULT}}${{env.CACHE_SUFFIX}}
- restore-keys: ccache${{env.CACHE_PREFIX_DEFAULT}}
+ key: ccache:v${{env.PG_MAJOR_VERSION}}${{env.CACHE_PREFIX_DEFAULT}}${{env.CACHE_SUFFIX}}
+ restore-keys: ccache:v${{env.PG_MAJOR_VERSION}}${{env.CACHE_PREFIX_DEFAULT}}
- &ccache_restore_branch_step
name: "ccache: Restore for branch ${{ github.ref_name }}"
@@ -315,8 +322,8 @@ jobs:
uses: actions/cache/restore@v5
with:
path: ${{ env.CCACHE_DIR }}
- key: ccache${{env.CACHE_PREFIX_BRANCH}}${{env.CACHE_SUFFIX}}
- restore-keys: ccache${{env.CACHE_PREFIX_BRANCH}}
+ key: ccache:v${{env.PG_MAJOR_VERSION}}${{env.CACHE_PREFIX_BRANCH}}${{env.CACHE_SUFFIX}}
+ restore-keys: ccache:v${{env.PG_MAJOR_VERSION}}${{env.CACHE_PREFIX_BRANCH}}
- &linux_prepare_workspace_step
name: Prepare workspace
diff --git a/src/tools/version_stamp.pl b/src/tools/version_stamp.pl
index 8f94e9e2886..c1891e34092 100755
--- a/src/tools/version_stamp.pl
+++ b/src/tools/version_stamp.pl
@@ -98,6 +98,9 @@ sed_file("configure.ac",
sed_file("meson.build",
qq{-e "/^project(/,/^)/ s/ version: '[0-9a-z.]*',/ version: '$fullversion',/"}
);
+sed_file(".github/workflows/pg-ci.yml",
+ "-e 's/^\\( *PG_MAJOR_VERSION: *\"\\)[0-9]*\"/\\1$majorversion\"/'"
+);
print "Stamped these files with version number $fullversion:\n$fixedfiles";
print "Don't forget to run autoconf $aconfver before committing.\n";
--
2.47.3
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: ci: namespace ccache by PostgreSQL major version
@ 2026-07-06 07:52 Peter Eisentraut <peter@eisentraut.org>
parent: Nazir Bilal Yavuz <byavuz81@gmail.com>
0 siblings, 2 replies; 19+ messages in thread
From: Peter Eisentraut @ 2026-07-06 07:52 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>; +Cc: Andres Freund <andres@anarazel.de>
On 03.07.26 16:26, Nazir Bilal Yavuz wrote:
> Hi,
>
> When the Postgres major version is updated, CI fails with [1]:
>
> 2026-07-02 17:01:01.938 UTC [828] FATAL: incompatible library
> "D:/a/postgresql/postgresql/build/tmp_install/usr/local/pgsql/lib/utf8_and_win.dll":
> version mismatch
> 2026-07-02 17:01:01.938 UTC [828] DETAIL: Server is version 20,
> library is version 19.
>
> This happens because the GitHub Actions ccache is restored across a
> version bump, so objects built against the previous version can be
> reused, producing libraries that no longer match the server version.
>
> Here is an attempt to solve this problem by namespacing ccache by
> Postgres major version. I added a 'PG_MAJOR_VERSION' variable to the
> CI file and used that as a prefix to ccache key. I made this
> 'PG_MAJOR_VERSION' variable automatically updated by
> 'version_stamp.pl' script.
>
> Another solution could be reading the version from build files (e.g
> meson.build), but then this read needs to be done at each CI run.
I'm suspicious about this direction. The major version is not the only
piece of data that determines ABI compatibility between the server and
extensions. This would only be a partial information. The ABI
information exists in the code, and so changes should be visible to
ccache. Maybe we are using ccache in the wrong mode or something (see
"depend mode", "direct mode", etc.).
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: ci: namespace ccache by PostgreSQL major version
@ 2026-07-06 08:05 Nazir Bilal Yavuz <byavuz81@gmail.com>
parent: Peter Eisentraut <peter@eisentraut.org>
1 sibling, 0 replies; 19+ messages in thread
From: Nazir Bilal Yavuz @ 2026-07-06 08:05 UTC (permalink / raw)
To: Peter Eisentraut <peter@eisentraut.org>; +Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>; Andres Freund <andres@anarazel.de>
Hi,
On Mon, 6 Jul 2026 at 10:52, Peter Eisentraut <peter@eisentraut.org> wrote:
>
> On 03.07.26 16:26, Nazir Bilal Yavuz wrote:
> > Hi,
> >
> > This happens because the GitHub Actions ccache is restored across a
> > version bump, so objects built against the previous version can be
> > reused, producing libraries that no longer match the server version.
> >
> > Here is an attempt to solve this problem by namespacing ccache by
> > Postgres major version.
>
> I'm suspicious about this direction. The major version is not the only
> piece of data that determines ABI compatibility between the server and
> extensions. This would only be a partial information. The ABI
> information exists in the code, and so changes should be visible to
> ccache. Maybe we are using ccache in the wrong mode or something (see
> "depend mode", "direct mode", etc.).
Thanks for the feedback!
Yes, you are right. I realized that after doing the further research
(I should have done that sooner, before sending the patch).
--
Regards,
Nazir Bilal Yavuz
Microsoft
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: ci: namespace ccache by PostgreSQL major version
@ 2026-07-07 17:35 Andres Freund <andres@anarazel.de>
parent: Peter Eisentraut <peter@eisentraut.org>
1 sibling, 1 reply; 19+ messages in thread
From: Andres Freund @ 2026-07-07 17:35 UTC (permalink / raw)
To: Peter Eisentraut <peter@eisentraut.org>; +Cc: Nazir Bilal Yavuz <byavuz81@gmail.com>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Hi,
On 2026-07-06 09:52:19 +0200, Peter Eisentraut wrote:
> On 03.07.26 16:26, Nazir Bilal Yavuz wrote:
> > Hi,
> >
> > When the Postgres major version is updated, CI fails with [1]:
> >
> > 2026-07-02 17:01:01.938 UTC [828] FATAL: incompatible library
> > "D:/a/postgresql/postgresql/build/tmp_install/usr/local/pgsql/lib/utf8_and_win.dll":
> > version mismatch
> > 2026-07-02 17:01:01.938 UTC [828] DETAIL: Server is version 20,
> > library is version 19.
> >
> > This happens because the GitHub Actions ccache is restored across a
> > version bump, so objects built against the previous version can be
> > reused, producing libraries that no longer match the server version.
> >
> > Here is an attempt to solve this problem by namespacing ccache by
> > Postgres major version. I added a 'PG_MAJOR_VERSION' variable to the
> > CI file and used that as a prefix to ccache key. I made this
> > 'PG_MAJOR_VERSION' variable automatically updated by
> > 'version_stamp.pl' script.
> >
> > Another solution could be reading the version from build files (e.g
> > meson.build), but then this read needs to be done at each CI run.
>
> I'm suspicious about this direction. The major version is not the only
> piece of data that determines ABI compatibility between the server and
> extensions. This would only be a partial information. The ABI information
> exists in the code, and so changes should be visible to ccache. Maybe we
> are using ccache in the wrong mode or something (see "depend mode", "direct
> mode", etc.).
Yea, I don't think that's the right answer. I'm reasonably sure I debugged
this issue a while ago:
https://postgr.es/m/phsrssp75npoyalqsolcd7fmnmlbzbmquc2p7w7mqjlw7432jk%40bzskz3luyjvb
Not a lot has happened on the ccache front, so I suspect we should start
adding the -fpch-deps flag I mentioned in that thread.
We could make adding -fpch-deps conditional on pch support being enabled, but
given that it doesn't do anything when pch is not used, I'd just add it when
supported by the compiler.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: ci: namespace ccache by PostgreSQL major version
@ 2026-07-16 21:27 Andres Freund <andres@anarazel.de>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 2 replies; 19+ messages in thread
From: Andres Freund @ 2026-07-16 21:27 UTC (permalink / raw)
To: Peter Eisentraut <peter@eisentraut.org>; +Cc: Nazir Bilal Yavuz <byavuz81@gmail.com>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Hi,
On 2026-07-07 13:35:34 -0400, Andres Freund wrote:
> On 2026-07-06 09:52:19 +0200, Peter Eisentraut wrote:
> > On 03.07.26 16:26, Nazir Bilal Yavuz wrote:
> > > Hi,
> > >
> > > When the Postgres major version is updated, CI fails with [1]:
> > >
> > > 2026-07-02 17:01:01.938 UTC [828] FATAL: incompatible library
> > > "D:/a/postgresql/postgresql/build/tmp_install/usr/local/pgsql/lib/utf8_and_win.dll":
> > > version mismatch
> > > 2026-07-02 17:01:01.938 UTC [828] DETAIL: Server is version 20,
> > > library is version 19.
> > >
> > > This happens because the GitHub Actions ccache is restored across a
> > > version bump, so objects built against the previous version can be
> > > reused, producing libraries that no longer match the server version.
> > >
> > > Here is an attempt to solve this problem by namespacing ccache by
> > > Postgres major version. I added a 'PG_MAJOR_VERSION' variable to the
> > > CI file and used that as a prefix to ccache key. I made this
> > > 'PG_MAJOR_VERSION' variable automatically updated by
> > > 'version_stamp.pl' script.
> > >
> > > Another solution could be reading the version from build files (e.g
> > > meson.build), but then this read needs to be done at each CI run.
> >
> > I'm suspicious about this direction. The major version is not the only
> > piece of data that determines ABI compatibility between the server and
> > extensions. This would only be a partial information. The ABI information
> > exists in the code, and so changes should be visible to ccache. Maybe we
> > are using ccache in the wrong mode or something (see "depend mode", "direct
> > mode", etc.).
>
> Yea, I don't think that's the right answer. I'm reasonably sure I debugged
> this issue a while ago:
>
> https://postgr.es/m/phsrssp75npoyalqsolcd7fmnmlbzbmquc2p7w7mqjlw7432jk%40bzskz3luyjvb
>
> Not a lot has happened on the ccache front, so I suspect we should start
> adding the -fpch-deps flag I mentioned in that thread.
>
> We could make adding -fpch-deps conditional on pch support being enabled, but
> given that it doesn't do anything when pch is not used, I'd just add it when
> supported by the compiler.
Attached is the change I propose.
Greetings,
Andres
Attachments:
[text/x-diff] v2-0001-meson-Fix-ccache-issues-when-using-precompiled-he.patch (1.9K, ../../gch4hgdyxnrlqsndn5qln4frftabwtscul5fvqhjpv34tu6ool@gfblk57l7j6e/2-v2-0001-meson-Fix-ccache-issues-when-using-precompiled-he.patch)
download | inline diff:
From 77df912a3172bbd3ea672708469d02ca938f79ee Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 15 Jul 2026 15:27:16 -0400
Subject: [PATCH v2] meson: Fix ccache issues when using precompiled headers
with gcc
Unfortunately the combination of gcc, precompiled headers, ccache and meson
currently is not safe without further options. The dependencies emitted by gcc
are insufficient to trigger rebuilds when headers "below" the precompiled
headers are changed. Whether that's a ccache, gcc or meson bug is
debatable. Luckily gcc's -fpch-deps option fixes the issue.
This problem occasionally leads to build failures, e.g. if only c.h,
postgres.h or pg_config_manual.h change. That's e.g. the case when creating a
new major version branch.
Discussion: https://postgr.es/m/CAN55FZ0tqR6Xz%3DiVFLc1BBoLOEHU775ARhcGYwggHA3XLA%3DoQg%40mail.gmail.com
Discussion: https://postgr.es/m/CA+hUKG+s7Yvt0PUnSQUEjCjysV-7-51n9B1h468Le3VJi0x4ZQ@mail.gmail.com
Discussion: https://postgr.es/m/phsrssp75npoyalqsolcd7fmnmlbzbmquc2p7w7mqjlw7432jk@bzskz3luyjvb
Discussion: https://github.com/ccache/ccache/issues/1686
Backpatch-through: 16, where meson support was added
---
meson.build | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/meson.build b/meson.build
index 61b5681851e..f4cde249242 100644
--- a/meson.build
+++ b/meson.build
@@ -2185,6 +2185,12 @@ common_functional_flags = [
# Disable optimizations that assume no overflow; needed for gcc 4.3+
'-fwrapv',
'-fexcess-precision=standard',
+ # Without -fpch-deps gcc emits dependencies that are insufficient for ccache
+ # to trigger a rebuild when the precompiled header changes. We could make
+ # this depend on using gcc and precompiled headers being enabled, but that's
+ # probably not worth it. See also
+ # https://github.com/ccache/ccache/issues/1686
+ '-fpch-deps',
]
cflags += cc.get_supported_arguments(common_functional_flags)
--
2.54.0.450.g9ac3f193c0
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: ci: namespace ccache by PostgreSQL major version
@ 2026-07-16 22:19 Jelte Fennema-Nio <postgres@jeltef.nl>
parent: Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 19+ messages in thread
From: Jelte Fennema-Nio @ 2026-07-16 22:19 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Peter Eisentraut <peter@eisentraut.org>; Nazir Bilal Yavuz <byavuz81@gmail.com>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Thu, 16 Jul 2026 at 23:27, Andres Freund <andres@anarazel.de> wrote:
> Attached is the change I propose.
Seems reasonable (assuming it indeed fixes the issue)
Regardless, I think we should still scope the caches on PG version
though for a completely different reason: Any backpatches (currently
only to 19) will become less and less cached as PG20 diverges from
PG19 if we have a shared cache. We should in addition make sure that
we create shared cached from REL_XX_STABLE branches and generate their
cache keys in a way that other branches restore them.
To be clear, this would look quite different from Bilal's patch.
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: ci: namespace ccache by PostgreSQL major version
@ 2026-07-17 08:57 Nazir Bilal Yavuz <byavuz81@gmail.com>
parent: Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 19+ messages in thread
From: Nazir Bilal Yavuz @ 2026-07-17 08:57 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Peter Eisentraut <peter@eisentraut.org>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Hi,
On Fri, 17 Jul 2026 at 00:27, Andres Freund <andres@anarazel.de> wrote:
>
> On 2026-07-07 13:35:34 -0400, Andres Freund wrote:
> >
> > Yea, I don't think that's the right answer. I'm reasonably sure I debugged
> > this issue a while ago:
> >
> > https://postgr.es/m/phsrssp75npoyalqsolcd7fmnmlbzbmquc2p7w7mqjlw7432jk%40bzskz3luyjvb
> >
> > Not a lot has happened on the ccache front, so I suspect we should start
> > adding the -fpch-deps flag I mentioned in that thread.
> >
> > We could make adding -fpch-deps conditional on pch support being enabled, but
> > given that it doesn't do anything when pch is not used, I'd just add it when
> > supported by the compiler.
>
> Attached is the change I propose.
Thanks for the patch! LGTM and I confirm it fixes the problem.
--
Regards,
Nazir Bilal Yavuz
Microsoft
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: ci: namespace ccache by PostgreSQL major version
@ 2026-07-17 09:41 Nazir Bilal Yavuz <byavuz81@gmail.com>
parent: Jelte Fennema-Nio <postgres@jeltef.nl>
0 siblings, 1 reply; 19+ messages in thread
From: Nazir Bilal Yavuz @ 2026-07-17 09:41 UTC (permalink / raw)
To: Jelte Fennema-Nio <postgres@jeltef.nl>; +Cc: Andres Freund <andres@anarazel.de>; Peter Eisentraut <peter@eisentraut.org>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Hi,
On Fri, 17 Jul 2026 at 01:19, Jelte Fennema-Nio <postgres@jeltef.nl> wrote:
>
> Regardless, I think we should still scope the caches on PG version
> though for a completely different reason: Any backpatches (currently
> only to 19) will become less and less cached as PG20 diverges from
> PG19 if we have a shared cache. We should in addition make sure that
> we create shared cached from REL_XX_STABLE branches and generate their
> cache keys in a way that other branches restore them.
AFAIK, if the cache hit ratio is below some point
(gha_ccache_decide.py -> 80 for branches other than master), the
branch regenerates its own cache. So, this shouldn't be a problem.
--
Regards,
Nazir Bilal Yavuz
Microsoft
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: ci: namespace ccache by PostgreSQL major version
@ 2026-07-17 11:50 Jelte Fennema-Nio <postgres@jeltef.nl>
parent: Nazir Bilal Yavuz <byavuz81@gmail.com>
0 siblings, 1 reply; 19+ messages in thread
From: Jelte Fennema-Nio @ 2026-07-17 11:50 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; Peter Eisentraut <peter@eisentraut.org>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Fri, 17 Jul 2026 at 11:41, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
> AFAIK, if the cache hit ratio is below some point
> (gha_ccache_decide.py -> 80 for branches other than master), the
> branch regenerates its own cache. So, this shouldn't be a problem.
Sure, but it would be nice to be able to create a shared cache by
pushing REL_19_STABLE if you're backporting multiple things.
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: ci: namespace ccache by PostgreSQL major version
@ 2026-07-17 13:06 Andres Freund <andres@anarazel.de>
parent: Jelte Fennema-Nio <postgres@jeltef.nl>
0 siblings, 2 replies; 19+ messages in thread
From: Andres Freund @ 2026-07-17 13:06 UTC (permalink / raw)
To: Jelte Fennema-Nio <postgres@jeltef.nl>; +Cc: Nazir Bilal Yavuz <byavuz81@gmail.com>; Peter Eisentraut <peter@eisentraut.org>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Hi,
On 2026-07-17 13:50:30 +0200, Jelte Fennema-Nio wrote:
> On Fri, 17 Jul 2026 at 11:41, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
> > AFAIK, if the cache hit ratio is below some point
> > (gha_ccache_decide.py -> 80 for branches other than master), the
> > branch regenerates its own cache. So, this shouldn't be a problem.
>
> Sure, but it would be nice to be able to create a shared cache by
> pushing REL_19_STABLE if you're backporting multiple things.
That's how it works. We restore the cache for master and for the current
branch. We save the cache, after pruning it, for the current branch iff the
hit rate was < 80%.
It'd be nice if we could restore the cache for the "base branch" of a branch,
instead of master, but that's not currently possible in GHA, from what I can
tell. Only caches on the default branch are accessible to other branches.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: ci: namespace ccache by PostgreSQL major version
@ 2026-07-17 15:27 Jacob Champion <jacob.champion@enterprisedb.com>
parent: Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 19+ messages in thread
From: Jacob Champion @ 2026-07-17 15:27 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Jelte Fennema-Nio <postgres@jeltef.nl>; Nazir Bilal Yavuz <byavuz81@gmail.com>; Peter Eisentraut <peter@eisentraut.org>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Fri, Jul 17, 2026 at 6:06 AM Andres Freund <andres@anarazel.de> wrote:
> That's how it works. We restore the cache for master and for the current
> branch. We save the cache, after pruning it, for the current branch iff the
> hit rate was < 80%.
>
> It'd be nice if we could restore the cache for the "base branch" of a branch,
> instead of master, but that's not currently possible in GHA, from what I can
> tell. Only caches on the default branch are accessible to other branches.
Quick tangent: is there a reason we put the refname into the cache
key? It seems like some of the logic there would happen by default if
we used the same cache prefix for all branches, since I thought GitHub
would perform the fallback-to-default-branch for us.
--Jacob
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: ci: namespace ccache by PostgreSQL major version
@ 2026-07-17 15:27 Andres Freund <andres@anarazel.de>
parent: Nazir Bilal Yavuz <byavuz81@gmail.com>
0 siblings, 0 replies; 19+ messages in thread
From: Andres Freund @ 2026-07-17 15:27 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: Peter Eisentraut <peter@eisentraut.org>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Hi,
On 2026-07-17 11:57:18 +0300, Nazir Bilal Yavuz wrote:
> On Fri, 17 Jul 2026 at 00:27, Andres Freund <andres@anarazel.de> wrote:
> >
> > On 2026-07-07 13:35:34 -0400, Andres Freund wrote:
> > >
> > > Yea, I don't think that's the right answer. I'm reasonably sure I debugged
> > > this issue a while ago:
> > >
> > > https://postgr.es/m/phsrssp75npoyalqsolcd7fmnmlbzbmquc2p7w7mqjlw7432jk%40bzskz3luyjvb
> > >
> > > Not a lot has happened on the ccache front, so I suspect we should start
> > > adding the -fpch-deps flag I mentioned in that thread.
> > >
> > > We could make adding -fpch-deps conditional on pch support being enabled, but
> > > given that it doesn't do anything when pch is not used, I'd just add it when
> > > supported by the compiler.
> >
> > Attached is the change I propose.
>
> Thanks for the patch! LGTM and I confirm it fixes the problem.
Thanks for reviewing. Pushed.
Greetings,
Andres
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: ci: namespace ccache by PostgreSQL major version
@ 2026-07-17 15:38 Andres Freund <andres@anarazel.de>
parent: Jacob Champion <jacob.champion@enterprisedb.com>
0 siblings, 1 reply; 19+ messages in thread
From: Andres Freund @ 2026-07-17 15:38 UTC (permalink / raw)
To: Jacob Champion <jacob.champion@enterprisedb.com>; +Cc: Jelte Fennema-Nio <postgres@jeltef.nl>; Nazir Bilal Yavuz <byavuz81@gmail.com>; Peter Eisentraut <peter@eisentraut.org>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Hi,
On 2026-07-17 08:27:39 -0700, Jacob Champion wrote:
> On Fri, Jul 17, 2026 at 6:06 AM Andres Freund <andres@anarazel.de> wrote:
> > That's how it works. We restore the cache for master and for the current
> > branch. We save the cache, after pruning it, for the current branch iff the
> > hit rate was < 80%.
> >
> > It'd be nice if we could restore the cache for the "base branch" of a branch,
> > instead of master, but that's not currently possible in GHA, from what I can
> > tell. Only caches on the default branch are accessible to other branches.
>
> Quick tangent: is there a reason we put the refname into the cache
> key? It seems like some of the logic there would happen by default if
> we used the same cache prefix for all branches, since I thought GitHub
> would perform the fallback-to-default-branch for us.
For cfbot it's kinda important to have *both* the master and branch specific
caches for a run. Often, after a rebase of a branch to a newer master, the
branch specific key will have a lower hit rate than during the last run, but
master's cache applies to a lot of the part of the build not modified by the
branch.
I also have some hope that eventually github will allow using caches from more
than just the default branch. For testing to-be-backpatched bug fixes - on fix
specific, per-major branches, it's pretty painful that they can only start
with master's cache.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: ci: namespace ccache by PostgreSQL major version
@ 2026-07-17 15:55 Jelte Fennema-Nio <postgres@jeltef.nl>
parent: Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 19+ messages in thread
From: Jelte Fennema-Nio @ 2026-07-17 15:55 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Nazir Bilal Yavuz <byavuz81@gmail.com>; Peter Eisentraut <peter@eisentraut.org>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Fri Jul 17, 2026 at 3:06 PM CEST, Andres Freund wrote:
> It'd be nice if we could restore the cache for the "base branch" of a branch,
> instead of master, but that's not currently possible in GHA, from what I can
> tell. Only caches on the default branch are accessible to other branches.
Hmm... The GHA permission model seems to be more complex every time I
look into it.... It's indeed not possible to restore caches from
non-default branches, *except* if the workflow was triggered by a PR and
that PR tries to restore a cache from the base branch of the PR.
So attached is something that makes this work as long as you open a PR
against the respective release branch. The more complex logic is
actually the part that cancels the branch build to make sure that you
don't run push and PR workflows at the same time.
After implementing that I think there's another approach possible too:
Allow users to trigger a cache build workflow *from the master branch*
that builds the cache for specific release branches (by checking them
out and building them).
Attachments:
[text/x-patch] 0001-Trigger-CI-on-PRs-and-restore-cache-from-their-base-.patch (14.1K, ../../DK0YVTWK4BW0.22LT8AQ6QZHIR@jeltef.nl/2-0001-Trigger-CI-on-PRs-and-restore-cache-from-their-base-.patch)
download | inline diff:
From 158e44c223ce076ce169f566b96a7e0c035826ca Mon Sep 17 00:00:00 2001
From: Jelte Fennema-Nio <postgres@jeltef.nl>
Date: Fri, 17 Jul 2026 16:22:03 +0200
Subject: [PATCH] Trigger CI on PRs and restore cache from their base branches
This makes PR CI builds work and makes them restore caches from their
base branches. One problem is that Github will trigger both a push and a
pull_request event if you push to a PR. This solves that by having the
push build skip itself if it sees that a PR is open. The problem is that
that doesn't work for pushes that are already running. So in addition
this adds a dummy workflow that cancels any push build on a branch for
which a PR was opened.
---
.github/workflows/pg-ci-cancel-superseded.yml | 59 ++++++++++
.github/workflows/pg-ci.yml | 111 ++++++++++++++++--
src/tools/ci/README | 24 ++++
3 files changed, 183 insertions(+), 11 deletions(-)
create mode 100644 .github/workflows/pg-ci-cancel-superseded.yml
diff --git a/.github/workflows/pg-ci-cancel-superseded.yml b/.github/workflows/pg-ci-cancel-superseded.yml
new file mode 100644
index 00000000000..49e4a3479ad
--- /dev/null
+++ b/.github/workflows/pg-ci-cancel-superseded.yml
@@ -0,0 +1,59 @@
+# Companion workflow to pg-ci.yml.
+#
+# When a PR is opened for an already-pushed branch, a push run for that
+# branch may still be executing: the skip check in pg-ci.yml's `setup` job
+# only helps push runs that start while the PR already exists. This workflow
+# cancels such a push run by entering the push runs' concurrency group with
+# cancel-in-progress enabled. Using a separate workflow, rather than a job
+# in pg-ci.yml, has two advantages:
+#
+# - a pull_request run of pg-ci.yml needs its own concurrency group, and a
+# workflow run can only be in one group
+# - workflow level concurrency takes effect when the run is created, before
+# waiting for a runner, so the superseded run is cancelled immediately
+#
+# Note that the cancelled push run will show up as a failed ('cancelled')
+# check on the PR's head commit, next to the pull_request run's results.
+# Cleaning that up would require a job with `actions: write` permissions
+# deleting the cancelled run, which doesn't seem worth it.
+#
+# This only triggers when a PR is (re)opened. Pushes to a branch with an
+# open PR skip themselves, see pg-ci.yml's `setup`; not triggering for them
+# also avoids cancelling those (quickly, successfully self-skipping) push
+# runs, which would make them show up as 'cancelled' instead.
+#
+# Pushes to master & the stable branches use a unique concurrency group per
+# run, so this workflow can never cancel those, even if a PR is opened with
+# one of them as the head branch.
+
+name: Cancel superseded push run
+
+on:
+ pull_request:
+ types: [opened, reopened]
+
+permissions: {}
+
+concurrency:
+ # This must render identically to pg-ci.yml's concurrency group for a
+ # push run of this PR's head branch.
+ group: |
+ pg-ci-refs/heads/${{ github.head_ref }}
+ cancel-in-progress: true
+
+jobs:
+ # The cancellation happens as a side effect of creating this workflow
+ # run: concurrency is processed when the run enters its group, before any
+ # job is scheduled. A workflow merely needs at least one job to be valid;
+ # it never has to execute, so this run finishes immediately without ever
+ # occupying a runner.
+ noop:
+ name: Cancelled superseded push run
+ # Constantly false, as this workflow only has pull_request triggers.
+ # Spelled this way instead of a literal `false` because actionlint
+ # flags constant conditions as likely mistakes, and it supports no
+ # inline ignore comments.
+ if: github.event_name != 'pull_request'
+ runs-on: ubuntu-slim
+ steps:
+ - run: 'true'
diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml
index 5bc5292d2a5..d7f0c416345 100644
--- a/.github/workflows/pg-ci.yml
+++ b/.github/workflows/pg-ci.yml
@@ -10,10 +10,19 @@ name: CI for PostgreSQL
on:
push:
- # TODO: It might make sense to also add PR based triggers, to make it easier
- # to use PRs on one's own repo, but it's a tad more complicated than just
- # adding the 'pull_request' event, as naively doing so would often lead to
- # running CI twice.
+ # A pull_request run can, unlike a push run, restore caches created on the
+ # PR's base branch. That makes opening a PR within one's own repository
+ # useful when working on top of a branch other than the default one (e.g.
+ # when backpatching), as the base branch's ccache is shared that way. See
+ # CACHE_PREFIX_BRANCH below.
+ #
+ # Once a branch has an open PR in the same repository, each push to it
+ # triggers both a push and a pull_request run. To avoid running CI twice,
+ # the `setup` job detects that case and disables all jobs of the push run,
+ # in favor of the strictly more useful pull_request run. A push run that
+ # is already executing when the PR is opened is cancelled by the
+ # companion workflow pg-ci-cancel-superseded.yml instead.
+ pull_request:
# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo
# contents during checkout.
@@ -26,8 +35,18 @@ concurrency:
# neither want to wait for prior runs, nor to cancel them, so that each
# separately pushed commit is tested. We achieve that by setting a unique
# concurrency group when on such a branch.
+ #
+ # For pull_request runs github.ref is the PR's merge ref
+ # (refs/pull/<nr>/merge), so each PR gets its own group, like a branch.
+ # This group intentionally never mixes push and pull_request runs;
+ # deduplication between those happens through the skip check in the
+ # `setup` job and through pg-ci-cancel-superseded.yml, which deliberately
+ # enters a push run's group to cancel it. The group uses a hardcoded
+ # 'pg-ci' prefix rather than the workflow name, so that renaming the
+ # workflow can't silently break that; changing the group's format still
+ # requires adjusting pg-ci-cancel-superseded.yml too.
group: |
- ${{github.workflow }}-${{
+ pg-ci-${{
case(github.ref == 'refs/heads/master' ||
(startsWith(github.ref, 'refs/heads/REL_') && endsWith(github.ref, '_STABLE')),
github.run_id,
@@ -134,15 +153,26 @@ env:
# A few variables to make expressions later on shorter
###
+ # Used by gha_ccache_decide.py to pick the target cache hit rate. This is
+ # intentionally false for pull_request runs, even those targeting the
+ # default branch: caches saved by a pull_request run are only restorable
+ # by later runs of the same PR, they are not shared with everything the
+ # way caches created on the default branch itself are.
ON_DEFAULT_BRANCH: ${{github.event.repository.default_branch == github.ref_name }}
# Note that we need to be careful to use a separator that can't be in branch
# names, otherwise e.g. caches for 'master' might be restored on the
# 'master-pending' branch.
+ #
+ # For pull_request runs use the PR's base branch instead of
+ # github.ref_name: the latter would be the meaningless "<nr>/merge", and
+ # the base branch's caches are exactly the ones a pull_request run is
+ # additionally permitted to restore (the reason to use PR based CI in the
+ # first place, see `on:` above).
CACHE_PREFIX_DEFAULT: >-
:${{ github.job }}:${{ github.event.repository.default_branch }}:
CACHE_PREFIX_BRANCH: >-
- :${{ github.job }}:${{ github.ref_name }}:
+ :${{ github.job }}:${{ github.base_ref || github.ref_name }}:
CACHE_SUFFIX: >-
${{ github.run_id }}:${{ github.run_attempt }}
@@ -193,6 +223,12 @@ jobs:
# if, none of it's depending tasks (i.e. the actual CI tasks) run either.
if: ${{vars.PG_CI_ENABLED == '1'}}
runs-on: *linux_runs_on
+ # The duplicate-run check below needs to list the repository's pull
+ # requests. Job level permissions replace the workflow level ones
+ # entirely, so contents: read has to be repeated here.
+ permissions:
+ contents: read
+ pull-requests: read
timeout-minutes: 1
outputs:
linux: ${{ steps.os.outputs.linux }}
@@ -220,14 +256,63 @@ jobs:
ulimit -a -H && ulimit -a -S
env
+ # Detect whether this push run is superseded by a pull_request run,
+ # see the comment at `on:` for why. Pushes to the default branch and
+ # the stable branches are never superseded: PRs are not used to manage
+ # those, and their runs create the caches other branches build on.
+ - name: Check for a superseding pull_request run
+ id: dup
+ if: github.event_name == 'push'
+ env:
+ GH_TOKEN: ${{ github.token }}
+ DEFAULT_BRANCH: ${{ github.event.repository.default_branch }}
+ shell: bash
+ run: |
+ skip=false
+ case "$GITHUB_REF_NAME" in
+ "$DEFAULT_BRANCH" | REL_*_STABLE)
+ ;;
+ *)
+ # Only PRs whose head branch is in this repository trigger
+ # pull_request runs of this workflow, so PRs from other forks
+ # targeting this repository must not suppress the push run. If
+ # the API call fails, assume there is no PR — better to run CI
+ # twice than not at all.
+ prs=$(gh pr list --repo "$GITHUB_REPOSITORY" --state open \
+ --head "$GITHUB_REF_NAME" \
+ --json headRepositoryOwner \
+ --jq '[.[] | select(.headRepositoryOwner.login == env.GITHUB_REPOSITORY_OWNER)] | length' \
+ || echo 0)
+ if [ "$prs" -gt 0 ]; then
+ skip=true
+ fi
+ ;;
+ esac
+ echo "skip=$skip" | tee -a "$GITHUB_OUTPUT"
+
- name: Parse ci-os-only
id: os
env:
MSG: ${{ github.event.head_commit.message }}
+ PR_HEAD_SHA: ${{ github.event.pull_request.head.sha }}
+ SUPERSEDED: ${{ steps.dup.outputs.skip }}
+ GH_TOKEN: ${{ github.token }}
shell: bash
run: |
all_os=${CI_OS_ONLY_JOBS}
- if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then
+ # pull_request payloads have no head_commit, fetch the head
+ # commit's message via the API instead (this job doesn't check out
+ # the repository, so we can't ask git). If the call fails, err on
+ # the side of running all jobs.
+ if [ -n "$PR_HEAD_SHA" ]; then
+ MSG=$(gh api "repos/$GITHUB_REPOSITORY/commits/$PR_HEAD_SHA" \
+ --jq .commit.message) || MSG=''
+ fi
+ if [ "$SUPERSEDED" = 'true' ]; then
+ sel=''
+ echo 'Superseded by the pull_request run for this branch, disabling all jobs' \
+ | tee -a "$GITHUB_STEP_SUMMARY"
+ elif printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then
sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1)
echo "ci-os-only selection: $sel"
else
@@ -294,15 +379,19 @@ jobs:
fetch-depth: ${{ env.CLONE_DEPTH }}
# We restore both the ccache from the default branch (typically master),
- # and from the current branch. This will often allow feature branches to
- # start out with a high cache hit ratio.
+ # and from the current branch (for pull_request runs: the PR's base
+ # branch, see CACHE_PREFIX_BRANCH). This will often allow feature
+ # branches to start out with a high cache hit ratio.
#
# With ccache it turns out to work to just restore two caches into the
# same directory, as it's basically a content addressed store. Stats
# could be corrupted, but we zero them out anyway.
- &ccache_restore_default_step
name: "ccache: Restore for default branch ${{github.event.repository.default_branch}}"
- if: ${{ env.ON_DEFAULT_BRANCH == 'false' }}
+ # Not worth restoring separately when the branch restore below uses
+ # the same key anyway, i.e. when on the default branch or in a
+ # pull_request run whose base is the default branch.
+ if: ${{ env.CACHE_PREFIX_BRANCH != env.CACHE_PREFIX_DEFAULT }}
uses: actions/cache/restore@v5
with:
path: ${{ env.CCACHE_DIR }}
@@ -310,7 +399,7 @@ jobs:
restore-keys: ccache${{env.CACHE_PREFIX_DEFAULT}}
- &ccache_restore_branch_step
- name: "ccache: Restore for branch ${{ github.ref_name }}"
+ name: "ccache: Restore for branch ${{ github.base_ref || github.ref_name }}"
id: ccache-restore-branch
uses: actions/cache/restore@v5
with:
diff --git a/src/tools/ci/README b/src/tools/ci/README
index 642e518c296..d2d66a808eb 100644
--- a/src/tools/ci/README
+++ b/src/tools/ci/README
@@ -43,6 +43,30 @@ https://github.com/<username>/<reponame>/settings/variables/actions and
create a new repository variable named PG_CI_ENABLED, with the value 1.
+Using pull requests for CI
+==========================
+
+CI runs both for pushed branches and for pull requests opened within a
+repository (the main postgres repository does not use pull requests, but
+personal forks can).
+
+For branches based on the repository's default branch the two behave the
+same, and simply pushing the branch is the easiest way to get CI. However,
+GitHub Actions only allows a workflow run to restore caches created on the
+branch itself, on the default branch, and - only for pull request runs - on
+the pull request's base branch. Thus, when working on top of another branch,
+e.g. when backpatching to REL_*_STABLE branches, it is worth opening a pull
+request within one's own repository, targeting that branch: after pushing
+the base branch once to populate its cache, its compilation results (ccache)
+are reused, instead of building from scratch each time.
+
+When a branch has an open pull request in the same repository, a push to it
+triggers both a push and a pull_request workflow run. To avoid duplicated
+work, the push run detects this situation and skips all its jobs in favor
+of the pull_request run. Additionally, opening a pull request cancels a
+push run of its branch that is still executing at that moment.
+
+
Viewing CI results in a GitHub repository
=========================================
--
2.54.0
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: ci: namespace ccache by PostgreSQL major version
@ 2026-07-17 16:12 Jacob Champion <jacob.champion@enterprisedb.com>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 0 replies; 19+ messages in thread
From: Jacob Champion @ 2026-07-17 16:12 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Jelte Fennema-Nio <postgres@jeltef.nl>; Nazir Bilal Yavuz <byavuz81@gmail.com>; Peter Eisentraut <peter@eisentraut.org>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Fri, Jul 17, 2026 at 8:38 AM Andres Freund <andres@anarazel.de> wrote:
> For cfbot it's kinda important to have *both* the master and branch specific
> caches for a run. Often, after a rebase of a branch to a newer master, the
> branch specific key will have a lower hit rate than during the last run, but
> master's cache applies to a lot of the part of the build not modified by the
> branch.
Oh, the rebase use case is interesting to think about. Thanks for the
explanation!
--Jacob
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: ci: namespace ccache by PostgreSQL major version
@ 2026-07-17 16:14 Jelte Fennema-Nio <postgres@jeltef.nl>
parent: Jelte Fennema-Nio <postgres@jeltef.nl>
0 siblings, 1 reply; 19+ messages in thread
From: Jelte Fennema-Nio @ 2026-07-17 16:14 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Nazir Bilal Yavuz <byavuz81@gmail.com>; Peter Eisentraut <peter@eisentraut.org>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Fri Jul 17, 2026 at 5:55 PM CEST, Jelte Fennema-Nio wrote:
> After implementing that I think there's another approach possible too:
> Allow users to trigger a cache build workflow *from the master branch*
> that builds the cache for specific release branches (by checking them
> out and building them).
Something like the attached, all done Claude. I'm gonna enjoy my weekend
now though. Feel free to take this over.
Attachments:
[text/x-patch] 0001-POC-Allow-running-for-a-different-branch-from-the-ma.patch (9.5K, ../../DK0ZA2KYTYVT.1RKL2W5BLVOF1@jeltef.nl/2-0001-POC-Allow-running-for-a-different-branch-from-the-ma.patch)
download | inline diff:
From d2fb68ef96b89dabaf1e4dcc95f6d1980157d3ea Mon Sep 17 00:00:00 2001
From: Jelte Fennema-Nio <postgres@jeltef.nl>
Date: Fri, 17 Jul 2026 18:08:54 +0200
Subject: [PATCH] POC: Allow running for a different branch from the master
branch
This can be useful to populate the caches.
---
.github/workflows/pg-ci.yml | 81 +++++++++++++++++++++++++++++++++++--
src/tools/ci/README | 28 +++++++++++++
2 files changed, 105 insertions(+), 4 deletions(-)
diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml
index 5bc5292d2a5..efe71ac4d01 100644
--- a/.github/workflows/pg-ci.yml
+++ b/.github/workflows/pg-ci.yml
@@ -15,6 +15,26 @@ on:
# adding the 'pull_request' event, as naively doing so would often lead to
# running CI twice.
+ # Build the ccache for another branch and publish it in the dispatching
+ # branch's cache scope. GitHub Actions only lets a workflow run restore
+ # caches created on its own branch or on the default branch, so plain
+ # pushes of branches based on a stable branch normally can't reuse that
+ # stable branch's compilation results. But caches created on the default
+ # branch are readable from everywhere. Thus, dispatching this workflow on
+ # master with e.g. REL_19_STABLE as the input makes that branch's ccache
+ # available to all branches (see the 'Derive cache base branch' step for
+ # the restore side). The run simply executes all of CI on the checked out
+ # branch, so it doubles as a full test of that branch. Re-dispatch
+ # occasionally to refresh the cache as the branch progresses.
+ workflow_dispatch:
+ inputs:
+ branch:
+ description: >-
+ Branch to check out and build, instead of the branch the workflow
+ was dispatched on (e.g. REL_19_STABLE)
+ type: string
+ required: false
+
# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo
# contents during checkout.
permissions:
@@ -26,8 +46,15 @@ concurrency:
# neither want to wait for prior runs, nor to cancel them, so that each
# separately pushed commit is tested. We achieve that by setting a unique
# concurrency group when on such a branch.
+ #
+ # workflow_dispatch cache builds (see `on:`) are keyed by the branch they
+ # build, so that a re-dispatch for a branch replaces a still-running
+ # build of it, while builds for different branches run concurrently. The
+ # bare branch name cannot collide with the push groups, which use the
+ # full refs/heads/... ref.
group: |
${{github.workflow }}-${{
+ inputs.branch ||
case(github.ref == 'refs/heads/master' ||
(startsWith(github.ref, 'refs/heads/REL_') && endsWith(github.ref, '_STABLE')),
github.run_id,
@@ -139,10 +166,15 @@ env:
# Note that we need to be careful to use a separator that can't be in branch
# names, otherwise e.g. caches for 'master' might be restored on the
# 'master-pending' branch.
+ #
+ # For workflow_dispatch runs with a `branch` input, the key names that
+ # branch rather than the one the workflow was dispatched on: the cache
+ # describes the sources that were built, not the ref the run executed on
+ # (which determines the cache's scope, see `on:` above).
CACHE_PREFIX_DEFAULT: >-
:${{ github.job }}:${{ github.event.repository.default_branch }}:
CACHE_PREFIX_BRANCH: >-
- :${{ github.job }}:${{ github.ref_name }}:
+ :${{ github.job }}:${{ inputs.branch || github.ref_name }}:
CACHE_SUFFIX: >-
${{ github.run_id }}:${{ github.run_attempt }}
@@ -292,6 +324,35 @@ jobs:
uses: actions/checkout@v6
with:
fetch-depth: ${{ env.CLONE_DEPTH }}
+ # Empty for everything but workflow_dispatch runs with a `branch`
+ # input, and empty means: check out the triggering commit.
+ ref: ${{ inputs.branch || '' }}
+
+ # Determine which stable branch, if any, the checked out sources
+ # belong to, by parsing the version from meson.build (a 'devel'
+ # version means master). The ccache restore below uses this as an
+ # additional fallback: a stable branch's ccache can be published in
+ # the globally readable default branch cache scope by dispatching
+ # this workflow, see `on:` above. This is what makes plain pushes of
+ # branches based on a stable branch start out with that branch's
+ # cache.
+ - &derive_cache_base_step
+ name: Derive cache base branch
+ id: cache-base
+ shell: bash
+ run: |
+ version=$(sed -n "s/^ *version: '\([^']*\)'.*/\1/p" meson.build | head -n 1)
+ restore_key=''
+ case "$version" in
+ *devel | '')
+ ;;
+ *)
+ major=${version%%[!0-9]*}
+ restore_key="ccache:${GITHUB_JOB}:REL_${major}_STABLE:"
+ ;;
+ esac
+ echo "version: $version"
+ echo "restore-key=$restore_key" | tee -a "$GITHUB_OUTPUT"
# We restore both the ccache from the default branch (typically master),
# and from the current branch. This will often allow feature branches to
@@ -302,7 +363,11 @@ jobs:
# could be corrupted, but we zero them out anyway.
- &ccache_restore_default_step
name: "ccache: Restore for default branch ${{github.event.repository.default_branch}}"
- if: ${{ env.ON_DEFAULT_BRANCH == 'false' }}
+ # Skip when the branch restore below would restore the same cache
+ # anyway. Comparing the prefixes, rather than checking
+ # ON_DEFAULT_BRANCH, makes a workflow_dispatch cache build on the
+ # default branch bootstrap from the default branch's cache.
+ if: ${{ env.CACHE_PREFIX_BRANCH != env.CACHE_PREFIX_DEFAULT }}
uses: actions/cache/restore@v5
with:
path: ${{ env.CCACHE_DIR }}
@@ -310,13 +375,15 @@ jobs:
restore-keys: ccache${{env.CACHE_PREFIX_DEFAULT}}
- &ccache_restore_branch_step
- name: "ccache: Restore for branch ${{ github.ref_name }}"
+ name: "ccache: Restore for branch ${{ inputs.branch || github.ref_name }}"
id: ccache-restore-branch
uses: actions/cache/restore@v5
with:
path: ${{ env.CCACHE_DIR }}
key: ccache${{env.CACHE_PREFIX_BRANCH}}${{env.CACHE_SUFFIX}}
- restore-keys: ccache${{env.CACHE_PREFIX_BRANCH}}
+ restore-keys: |
+ ccache${{env.CACHE_PREFIX_BRANCH}}
+ ${{ steps.cache-base.outputs.restore-key }}
- &linux_prepare_workspace_step
name: Prepare workspace
@@ -493,6 +560,7 @@ jobs:
- *nix_sysinfo_step
- *checkout_step
+ - *derive_cache_base_step
- *ccache_restore_default_step
- *ccache_restore_branch_step
- *linux_prepare_workspace_step
@@ -555,6 +623,7 @@ jobs:
- *nix_sysinfo_step
- *checkout_step
+ - *derive_cache_base_step
- *ccache_restore_default_step
- *ccache_restore_branch_step
- *linux_prepare_workspace_step
@@ -645,6 +714,7 @@ jobs:
- *nix_sysinfo_step
- *checkout_step
+ - *derive_cache_base_step
- *ccache_restore_default_step
- *ccache_restore_branch_step
- *linux_prepare_workspace_step
@@ -795,6 +865,7 @@ jobs:
path: ${{ env.MACPORTS_CACHE }}
key: ${{ steps.mp-key.outputs.key }}
+ - *derive_cache_base_step
- *ccache_restore_default_step
- *ccache_restore_branch_step
@@ -1116,6 +1187,7 @@ jobs:
shell: cmd
run: mkdir ${{env.PG_REGRESS_SOCK_DIR}}
+ - *derive_cache_base_step
- *ccache_restore_default_step
- *ccache_restore_branch_step
@@ -1174,6 +1246,7 @@ jobs:
steps:
- *nix_sysinfo_step
- *checkout_step
+ - *derive_cache_base_step
- *ccache_restore_default_step
- *ccache_restore_branch_step
diff --git a/src/tools/ci/README b/src/tools/ci/README
index 642e518c296..24ba82ebbde 100644
--- a/src/tools/ci/README
+++ b/src/tools/ci/README
@@ -43,6 +43,34 @@ https://github.com/<username>/<reponame>/settings/variables/actions and
create a new repository variable named PG_CI_ENABLED, with the value 1.
+Building shared caches for stable branches
+==========================================
+
+GitHub Actions only allows a workflow run to restore compilation caches
+(ccache) created on the run's own branch or on the repository's default
+branch. A branch based on e.g. REL_19_STABLE therefore normally starts
+building from scratch (or from the default branch's cache, which mostly
+misses), even when REL_19_STABLE itself was recently built by CI. This
+makes iterating on backpatches unnecessarily slow.
+
+However, caches created by runs on the default branch are readable from
+every branch. To take advantage of that, the CI workflow can be manually
+dispatched on the default branch with the name of another branch as input:
+
+ gh workflow run pg-ci.yml --ref master -f branch=REL_19_STABLE
+
+Such a run checks out and builds the given branch, while storing the
+resulting caches in the default branch's cache scope. CI runs detect which
+stable branch the sources they're testing belong to, based on the version
+in meson.build, and restore that branch's shared cache if it exists. Since
+the run executes the whole CI workflow, it also functions as a full test of
+the given branch.
+
+Occasionally re-dispatching the workflow refreshes the cache as the stable
+branch progresses. Caches unused for 7 days are evicted by GitHub, so
+branches only being backpatched to rarely may need a re-dispatch anyway.
+
+
Viewing CI results in a GitHub repository
=========================================
--
2.54.0
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: ci: namespace ccache by PostgreSQL major version
@ 2026-07-20 20:46 Jelte Fennema-Nio <postgres@jeltef.nl>
parent: Jelte Fennema-Nio <postgres@jeltef.nl>
0 siblings, 1 reply; 19+ messages in thread
From: Jelte Fennema-Nio @ 2026-07-20 20:46 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Nazir Bilal Yavuz <byavuz81@gmail.com>; Peter Eisentraut <peter@eisentraut.org>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Fri Jul 17, 2026 at 6:14 PM CEST, Jelte Fennema-Nio wrote:
> Something like the attached, all done Claude. I'm gonna enjoy my weekend
> now though. Feel free to take this over.
Changed it a bit and tried it out. This works pretty well and seems like
a nice improvement for people that actually backpatch things. I think I
like this simplicity a lot better than the patch I submitted earlier to
run CI on PRs.
Attachments:
[text/x-patch] v2-0001-ci-Make-stable-branch-ccache-usable-by-branches-b.patch (10.7K, ../../DK3OXY9DSO0B.3HUAWS1MF0LOP@jeltef.nl/2-v2-0001-ci-Make-stable-branch-ccache-usable-by-branches-b.patch)
download | inline diff:
From 629ce282b36fb1ac28e273abf452427d3d3d8418 Mon Sep 17 00:00:00 2001
From: Jelte Fennema-Nio <postgres@jeltef.nl>
Date: Fri, 17 Jul 2026 18:08:54 +0200
Subject: [PATCH v2] ci: Make stable branch ccache usable by branches based on
them
GitHub Actions only lets a workflow run restore caches that were created
on the run's own branch or on the repository's default branch. A branch
based on e.g. REL_19_STABLE therefore starts compiling from scratch (or
from master's cache, which mostly misses for stable branch sources),
even when REL_19_STABLE itself was recently built by CI. That makes
iterating on backpatches unnecessarily slow.
Caches created on the default branch, however, are readable from every
branch. So this adds a `workflow_dispatch` trigger with a `branch`
input, which allows dispatching the workflow on master while checking
out and building another branch:
gh workflow run pg-ci.yml --ref master -f branch=REL_19_STABLE
Such a run publishes the resulting ccache in master's globally readable
cache scope, under keys naming the branch that was built (e.g.
`ccache:linux-meson-64:REL_19_STABLE:`). On the restore side, every CI
run now derives which stable branch its sources belong to from the
version in `meson.build`, and prefers that branch's shared cache over
master's when restoring the base cache. So after a single dispatch for
REL_19_STABLE, any push of a branch based on it starts out with a warm
cache instead of a cold one.
---
.github/workflows/pg-ci.yml | 90 +++++++++++++++++++++++++++++++++----
src/tools/ci/README | 24 ++++++++++
2 files changed, 106 insertions(+), 8 deletions(-)
diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml
index 5bc5292d2a5..3b682e4db6e 100644
--- a/.github/workflows/pg-ci.yml
+++ b/.github/workflows/pg-ci.yml
@@ -15,6 +15,26 @@ on:
# adding the 'pull_request' event, as naively doing so would often lead to
# running CI twice.
+ # Build the ccache for another branch and publish it in the dispatching
+ # branch's cache scope. GitHub Actions only lets a workflow run restore
+ # caches created on its own branch or on the default branch, so plain
+ # pushes of branches based on a stable branch normally can't reuse that
+ # stable branch's compilation results. But caches created on the default
+ # branch are readable from everywhere. Thus, dispatching this workflow on
+ # master with e.g. REL_19_STABLE as the input makes that branch's ccache
+ # available to all branches (see the 'Derive cache base branch' step for
+ # the restore side). The run simply executes all of CI on the checked out
+ # branch, so it doubles as a full test of that branch. Re-dispatch
+ # occasionally to refresh the cache as the branch progresses.
+ workflow_dispatch:
+ inputs:
+ branch:
+ description: >-
+ Branch to check out and build, instead of the branch the workflow
+ was dispatched on (e.g. REL_19_STABLE)
+ type: string
+ required: false
+
# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo
# contents during checkout.
permissions:
@@ -26,8 +46,15 @@ concurrency:
# neither want to wait for prior runs, nor to cancel them, so that each
# separately pushed commit is tested. We achieve that by setting a unique
# concurrency group when on such a branch.
+ #
+ # workflow_dispatch cache builds (see `on:`) are keyed by the branch they
+ # build, so that a re-dispatch for a branch replaces a still-running
+ # build of it, while builds for different branches run concurrently. The
+ # bare branch name cannot collide with the push groups, which use the
+ # full refs/heads/... ref.
group: |
${{github.workflow }}-${{
+ inputs.branch ||
case(github.ref == 'refs/heads/master' ||
(startsWith(github.ref, 'refs/heads/REL_') && endsWith(github.ref, '_STABLE')),
github.run_id,
@@ -139,10 +166,15 @@ env:
# Note that we need to be careful to use a separator that can't be in branch
# names, otherwise e.g. caches for 'master' might be restored on the
# 'master-pending' branch.
+ #
+ # For workflow_dispatch runs with a `branch` input, the key names that
+ # branch rather than the one the workflow was dispatched on: the cache
+ # describes the sources that were built, not the ref the run executed on
+ # (which determines the cache's scope, see `on:` above).
CACHE_PREFIX_DEFAULT: >-
:${{ github.job }}:${{ github.event.repository.default_branch }}:
CACHE_PREFIX_BRANCH: >-
- :${{ github.job }}:${{ github.ref_name }}:
+ :${{ github.job }}:${{ inputs.branch || github.ref_name }}:
CACHE_SUFFIX: >-
${{ github.run_id }}:${{ github.run_attempt }}
@@ -292,25 +324,61 @@ jobs:
uses: actions/checkout@v6
with:
fetch-depth: ${{ env.CLONE_DEPTH }}
-
- # We restore both the ccache from the default branch (typically master),
- # and from the current branch. This will often allow feature branches to
- # start out with a high cache hit ratio.
+ # Empty for everything but workflow_dispatch runs with a `branch`
+ # input, and empty means: check out the triggering commit.
+ ref: ${{ inputs.branch || '' }}
+
+ # Determine which stable branch, if any, the checked out sources
+ # belong to, by parsing the version from meson.build (a 'devel'
+ # version means master). The base branch restore below prefers this
+ # over the default branch's cache: a stable branch's ccache can be
+ # published in the globally readable default branch cache scope by
+ # dispatching this workflow, see `on:` above. This is what makes
+ # plain pushes of branches based on a stable branch start out with
+ # that branch's cache, rather than master's, which mostly misses for
+ # stable branch sources.
+ - &derive_cache_base_step
+ name: Derive cache base branch
+ id: cache-base
+ shell: bash
+ run: |
+ version=$(sed -n "s/^ *version: '\([^']*\)'.*/\1/p" meson.build | head -n 1)
+ restore_key=''
+ case "$version" in
+ *devel | '')
+ ;;
+ *)
+ major=${version%%[!0-9]*}
+ restore_key="ccache:${GITHUB_JOB}:REL_${major}_STABLE:"
+ ;;
+ esac
+ echo "version: $version"
+ echo "restore-key=$restore_key" | tee -a "$GITHUB_OUTPUT"
+
+ # We restore both the ccache from the base branch (the stable branch
+ # the sources belong to if its shared cache exists, the default
+ # branch otherwise), and from the current branch. This will often
+ # allow feature branches to start out with a high cache hit ratio.
#
# With ccache it turns out to work to just restore two caches into the
# same directory, as it's basically a content addressed store. Stats
# could be corrupted, but we zero them out anyway.
- &ccache_restore_default_step
- name: "ccache: Restore for default branch ${{github.event.repository.default_branch}}"
+ name: "ccache: Restore for base branch"
if: ${{ env.ON_DEFAULT_BRANCH == 'false' }}
uses: actions/cache/restore@v5
with:
path: ${{ env.CCACHE_DIR }}
key: ccache${{env.CACHE_PREFIX_DEFAULT}}${{env.CACHE_SUFFIX}}
- restore-keys: ccache${{env.CACHE_PREFIX_DEFAULT}}
+ # restore-keys are tried in order and only the first match is
+ # restored, so the derived stable branch cache (empty entries are
+ # ignored) takes precedence over the default branch's.
+ restore-keys: |
+ ${{ steps.cache-base.outputs.restore-key }}
+ ccache${{env.CACHE_PREFIX_DEFAULT}}
- &ccache_restore_branch_step
- name: "ccache: Restore for branch ${{ github.ref_name }}"
+ name: "ccache: Restore for branch ${{ inputs.branch || github.ref_name }}"
id: ccache-restore-branch
uses: actions/cache/restore@v5
with:
@@ -493,6 +561,7 @@ jobs:
- *nix_sysinfo_step
- *checkout_step
+ - *derive_cache_base_step
- *ccache_restore_default_step
- *ccache_restore_branch_step
- *linux_prepare_workspace_step
@@ -555,6 +624,7 @@ jobs:
- *nix_sysinfo_step
- *checkout_step
+ - *derive_cache_base_step
- *ccache_restore_default_step
- *ccache_restore_branch_step
- *linux_prepare_workspace_step
@@ -645,6 +715,7 @@ jobs:
- *nix_sysinfo_step
- *checkout_step
+ - *derive_cache_base_step
- *ccache_restore_default_step
- *ccache_restore_branch_step
- *linux_prepare_workspace_step
@@ -795,6 +866,7 @@ jobs:
path: ${{ env.MACPORTS_CACHE }}
key: ${{ steps.mp-key.outputs.key }}
+ - *derive_cache_base_step
- *ccache_restore_default_step
- *ccache_restore_branch_step
@@ -1116,6 +1188,7 @@ jobs:
shell: cmd
run: mkdir ${{env.PG_REGRESS_SOCK_DIR}}
+ - *derive_cache_base_step
- *ccache_restore_default_step
- *ccache_restore_branch_step
@@ -1174,6 +1247,7 @@ jobs:
steps:
- *nix_sysinfo_step
- *checkout_step
+ - *derive_cache_base_step
- *ccache_restore_default_step
- *ccache_restore_branch_step
diff --git a/src/tools/ci/README b/src/tools/ci/README
index 642e518c296..dcab23e02be 100644
--- a/src/tools/ci/README
+++ b/src/tools/ci/README
@@ -43,6 +43,30 @@ https://github.com/<username>/<reponame>/settings/variables/actions and
create a new repository variable named PG_CI_ENABLED, with the value 1.
+Building shared caches for stable branches
+==========================================
+
+GitHub Actions only allows a workflow run to restore compilation caches
+(ccache) created on the run's own branch or on the repository's default
+branch. A branch based on e.g. REL_19_STABLE therefore normally starts
+building from scratch (or from the default branch's cache, which mostly
+misses), even when REL_19_STABLE itself was recently built by CI. This
+makes iterating on backpatches unnecessarily slow.
+
+However, caches created by runs on the default branch are readable from
+every branch. To take advantage of that, the CI workflow can be manually
+dispatched on the default branch with the name of another branch as input:
+
+ gh workflow run pg-ci.yml --ref master -f branch=REL_19_STABLE
+
+Such a run checks out and builds the given branch, while storing the
+resulting caches in the default branch's cache scope. CI runs detect which
+stable branch the sources they're testing belong to, based on the version
+in meson.build, and restore that branch's shared cache if it exists. Since
+the run executes the whole CI workflow, it also functions as a full test of
+the given branch.
+
+
Viewing CI results in a GitHub repository
=========================================
--
2.54.0
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: ci: namespace ccache by PostgreSQL major version
@ 2026-07-21 11:26 Nazir Bilal Yavuz <byavuz81@gmail.com>
parent: Jelte Fennema-Nio <postgres@jeltef.nl>
0 siblings, 1 reply; 19+ messages in thread
From: Nazir Bilal Yavuz @ 2026-07-21 11:26 UTC (permalink / raw)
To: Jelte Fennema-Nio <postgres@jeltef.nl>; +Cc: Andres Freund <andres@anarazel.de>; Peter Eisentraut <peter@eisentraut.org>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Hi,
On Mon, 20 Jul 2026 at 23:46, Jelte Fennema-Nio <postgres@jeltef.nl> wrote:
>
> On Fri Jul 17, 2026 at 6:14 PM CEST, Jelte Fennema-Nio wrote:
> > Something like the attached, all done Claude. I'm gonna enjoy my weekend
> > now though. Feel free to take this over.
>
> Changed it a bit and tried it out. This works pretty well and seems like
> a nice improvement for people that actually backpatch things. I think I
> like this simplicity a lot better than the patch I submitted earlier to
> run CI on PRs.
I think this is a nice improvement for the people who backpatch things
and have feature branches.
Since this CI run will only run for updating the cache, I think
running tests etc. are not important and time consuming. What do you
think about disabling them for these types of CI runs; so they will
only build, then update the cache and done?
--
Regards,
Nazir Bilal Yavuz
Microsoft
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: ci: namespace ccache by PostgreSQL major version
@ 2026-07-21 11:54 Jelte Fennema-Nio <postgres@jeltef.nl>
parent: Nazir Bilal Yavuz <byavuz81@gmail.com>
0 siblings, 0 replies; 19+ messages in thread
From: Jelte Fennema-Nio @ 2026-07-21 11:54 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; Peter Eisentraut <peter@eisentraut.org>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Tue, 21 Jul 2026 at 13:26, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
> Since this CI run will only run for updating the cache, I think
> running tests etc. are not important and time consuming. What do you
> think about disabling them for these types of CI runs; so they will
> only build, then update the cache and done?
Sounds reasonable to me
^ permalink raw reply [nested|flat] 19+ messages in thread
end of thread, other threads:[~2026-07-21 11:54 UTC | newest]
Thread overview: 19+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-07-03 14:26 ci: namespace ccache by PostgreSQL major version Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-07-06 07:52 ` Peter Eisentraut <peter@eisentraut.org>
2026-07-06 08:05 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-07-07 17:35 ` Andres Freund <andres@anarazel.de>
2026-07-16 21:27 ` Andres Freund <andres@anarazel.de>
2026-07-16 22:19 ` Jelte Fennema-Nio <postgres@jeltef.nl>
2026-07-17 09:41 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-07-17 11:50 ` Jelte Fennema-Nio <postgres@jeltef.nl>
2026-07-17 13:06 ` Andres Freund <andres@anarazel.de>
2026-07-17 15:27 ` Jacob Champion <jacob.champion@enterprisedb.com>
2026-07-17 15:38 ` Andres Freund <andres@anarazel.de>
2026-07-17 16:12 ` Jacob Champion <jacob.champion@enterprisedb.com>
2026-07-17 15:55 ` Jelte Fennema-Nio <postgres@jeltef.nl>
2026-07-17 16:14 ` Jelte Fennema-Nio <postgres@jeltef.nl>
2026-07-20 20:46 ` Jelte Fennema-Nio <postgres@jeltef.nl>
2026-07-21 11:26 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-07-21 11:54 ` Jelte Fennema-Nio <postgres@jeltef.nl>
2026-07-17 08:57 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-07-17 15:27 ` Andres Freund <andres@anarazel.de>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox