public inbox for [email protected]  

help / color / mirror / Atom feed

From: Justin Pryzby <[email protected]>
Subject: [PATCH 05/11] cirrus/ccache: disable compression and show stats
Date: Sun, 3 Apr 2022 00:10:20 -0500

Since v4.0, ccache enables zstd compression by default, saving roughly
2x-3x.  But, cirrus caches are compressed as tar.gz, so we could disable
ccache compression, allowing cirrus to gzip the uncompressed data
(better than ccache's default of zstd-1).

With default compression enabled (https://cirrus-ci.com/task/6692342840164352):
linux/debian/bullseye has 4.2; 99MB cirrus cache; cache_size_kibibyte	109616
macos has 4.5.1: 47MB cirrus cache; cache_size_kibibyte	52500
freebsd has 3.7.12: 42MB cirrus cache; cache_size_kibibyte	134064
XXX windows has 4.7.2; 180MB cirrus cache; cache_size_kibibyte	51179
todo: compiler warnings

With compression disabled (https://cirrus-ci.com/task/4614182514458624):
linux: 91MB cirrus cache; cache_size_kibibyte	316136
macos: 41MB cirrus cache; cache_size_kibibyte	118068
windows: 42MB cirrus cache; cache_size_kibibyte	134064
freebsd is the same

The stats should be shown and/or logged.
ccache --show-stats shows the *cumulative* stats (including prior
compilations)
ccache --zero-stats clears out not only the global stats, but the
per-file cache stats (from which the global stats are derived) - which
obviously makes the cache work poorly.

Note that ccache 4.4 supports CCACHE_STATSLOG, which seems ideal.
The log should be written *outside* the ccache folder - it shouldn't be
preserved across cirrusci task invocations.

freebsd, linux, macos
ci-os-only: linux
---
 .cirrus.yml | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)

diff --git a/.cirrus.yml b/.cirrus.yml
index 47c4caf51ac..d9660149a73 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -16,7 +16,11 @@ env:
   # Useful to be able to analyse what in a script takes long
   CIRRUS_LOG_TIMESTAMP: true
 
-  CCACHE_MAXSIZE: "250M"
+  CCACHE_MAXSIZE: "750M"
+  CCACHE_HARDLINK: 1
+  #CCACHE_NOCOMPRESS: 1
+  # The statslog should be stored somewhere *outside* the persistent ccache dir
+  CCACHE_STATSLOG: ${CIRRUS_WORKING_DIR}/ccache-stats.log
 
   # target to test, for all but windows
   CHECK: check-world PROVE_FLAGS=$PROVE_FLAGS
@@ -236,7 +240,7 @@ task:
     - name: Linux - Debian Bullseye - Meson
 
       env:
-        CCACHE_MAXSIZE: "800M" # tests two different builds
+        CCACHE_MAXSIZE: "2400M" # tests two different builds
 
       configure_script: |
         su postgres <<-EOF
@@ -289,7 +293,7 @@ task:
 
       ccache_stats_end_script:
         ccache -s
-        CCACHE_MAXSIZE=400M ccache --cleanup
+        CCACHE_MAXSIZE=1200M ccache --cleanup
 
       on_failure:
         <<: *on_failure_meson
@@ -392,7 +396,7 @@ task:
       -DPG_TEST_EXTRA="$PG_TEST_EXTRA" \
       build
 
-  build_script: ninja -C build -j${BUILD_JOBS}
+  build_script: ninja -C build -j${BUILD_JOBS} && ccache --show-log-stats
   upload_caches: ccache
 
   test_world_script: |
@@ -499,7 +503,7 @@ task:
     # flag combinations.  The cache size is set high now, and manually pruned later,
     # to avoid cache cleaning cycles during the early compilations pruning
     # compilation results which could've been used during later compilations.
-    CCACHE_MAXSIZE: "2G"
+    CCACHE_MAXSIZE: "6G"
     CCACHE_DIR: "/tmp/ccache_dir"
 
     LINUX_CONFIGURE_FEATURES: *LINUX_CONFIGURE_FEATURES
@@ -620,7 +624,7 @@ task:
   always:
     ccache_stats_end_script:
       ccache -s
-      CCACHE_MAXSIZE=1G ccache --cleanup
+      CCACHE_MAXSIZE=3G ccache --cleanup
 
     upload_caches: ccache
 
-- 
2.25.1


--I4VOKWutKNZEOIPu
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
 filename="0006-cirrus-warnings-use-a-single-common-always-block.patch"

view thread (8+ messages)  latest in thread

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: [email protected]
  Cc: [email protected]
  Subject: Re: [PATCH 05/11] cirrus/ccache: disable compression and show stats
  In-Reply-To: <no-message-id-1859086@localhost>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox