public inbox for [email protected]  

help / color / mirror / Atom feed

From: Justin Pryzby <[email protected]>
Subject: [PATCH 7/8] cirrus/ccache: disable compression and show stats
Date: Sun, 3 Apr 2022 00:10:20 -0500

Since v4.0, ccache enables zstd compression by default, saving roughly
2x-3x.  But, cirrus caches are compressed as tar.gz, so we could disable
ccache compression, allowing cirrus to gzip the uncompressed data
(better than ccache's default of zstd-1).

With default compression enabled (https://cirrus-ci.com/task/6692342840164352):
linux/debian/bullseye has 4.2; 99MB cirrus cache; cache_size_kibibyte	109616
macos has 4.5.1: 47MB cirrus cache; cache_size_kibibyte	52500
freebsd has 3.7.12: 42MB cirrus cache; cache_size_kibibyte	134064
XXX windows has 4.7.2; 180MB cirrus cache; cache_size_kibibyte	51179
todo: compiler warnings

With compression disabled (https://cirrus-ci.com/task/4614182514458624):
linux: 91MB cirrus cache; cache_size_kibibyte	316136
macos: 41MB cirrus cache; cache_size_kibibyte	118068
windows: 42MB cirrus cache; cache_size_kibibyte	134064
freebsd is the same

The stats should be shown and/or logged.
ccache --show-stats shows the *cumulative* stats (including prior
compilations)
ccache --zero-stats clears out not only the global stats, but the
per-file cache stats (from which the global stats are derived) - which
obviously makes the cache work poorly.

Note that ccache 4.4 supports CCACHE_STATSLOG, which seems ideal.
The log should be written *outside* the ccache folder - it shouldn't be
preserved across cirrusci task invocations.

freebsd, linux, macos
ci-os-only: linux
---
 .cirrus.yml | 13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)

diff --git a/.cirrus.yml b/.cirrus.yml
index 221431e70c4..d1c3119e4ff 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -16,7 +16,9 @@ env:
   # Useful to be able to analyse what in a script takes long
   CIRRUS_LOG_TIMESTAMP: true
 
-  CCACHE_MAXSIZE: "250M"
+  CCACHE_MAXSIZE: "750M"
+  CCACHE_NOCOMPRESS: 1
+  #CCACHE_STATSLOG: ccache-stats.log
 
   # target to test, for all but windows
   CHECK: check-world PROVE_FLAGS=$PROVE_FLAGS
@@ -236,7 +238,7 @@ task:
     - name: Linux - Debian Bullseye - Meson
 
       env:
-        CCACHE_MAXSIZE: "400M" # tests two different builds
+        CCACHE_MAXSIZE: "1200M" # tests two different builds
 
       configure_script: |
         su postgres <<-EOF
@@ -307,6 +309,7 @@ task:
 
     CIRRUS_WORKING_DIR: ${HOME}/pgsql/
     CCACHE_DIR: ${HOME}/ccache
+    CCACHE_STATSLOG: ${CCACHE_DIR}.stats.log
     HOMEBREW_CACHE: ${HOME}/homebrew-cache
     PERL5LIB: ${HOME}/perl5/lib/perl5
 
@@ -388,7 +391,7 @@ task:
       -DPG_TEST_EXTRA="$PG_TEST_EXTRA" \
       build
 
-  build_script: ninja -C build -j${BUILD_JOBS}
+  build_script: ninja -C build -j${BUILD_JOBS} && ccache --show-log-stats
   upload_caches: ccache
 
   test_world_script: |
@@ -496,14 +499,14 @@ task:
 
     # Use larger ccache cache, as this task compiles with multiple compilers /
     # flag combinations
-    CCACHE_MAXSIZE: "1GB"
+    CCACHE_MAXSIZE: "3G"
     CCACHE_DIR: "/tmp/ccache_dir"
 
     LINUX_CONFIGURE_FEATURES: *LINUX_CONFIGURE_FEATURES
     LINUX_MESON_FEATURES: *LINUX_MESON_FEATURES
 
   # task that did not run, count as a success, so we need to recheck Linux'
-  # condition here ...
+  # condition here; cirus warns if the "only_if" condition doesn't match the task being depended on
   only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*linux.*'
 
   container:
-- 
2.25.1


--rPFbv4B2w2tcwGo5
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
 filename="0008-cirrus-warnings-use-a-single-common-always-block.patch"

view thread (11+ messages)  latest in thread

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: [email protected]
  Cc: [email protected]
  Subject: Re: [PATCH 7/8] cirrus/ccache: disable compression and show stats
  In-Reply-To: <no-message-id-1859060@localhost>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox