MIME-Version: 1.0
References: <ahh972Lord8jglyO@alvherre.pgsql>
 <ahiLmvtaFWnTsFVu@alvherre.pgsql>
 <3wulk62iw2c5n3bxnkdpumm4updcbw3id3ahwkq2566q7m7jbj@qkbdz4rrukbw>
In-Reply-To: <3wulk62iw2c5n3bxnkdpumm4updcbw3id3ahwkq2566q7m7jbj@qkbdz4rrukbw>
From: Jakub Wartak <jakub.wartak@enterprisedb.com>
Date: Fri, 29 May 2026 15:22:50 +0200
Message-ID: 
 <CAKZiRmwA6sMZqdumO1+_TmzyhqFAJ12Ufs-iRS1Zms=TndcYjQ@mail.gmail.com>
Subject: Re: Heads Up: cirrus-ci is shutting down June 1st
To: Andres Freund <andres@anarazel.de>
Cc: =?UTF-8?Q?=C3=81lvaro_Herrera?= <alvherre@kurilemu.de>,
	Nazir Bilal Yavuz <byavuz81@gmail.com>,
 Jelte Fennema-Nio <postgres@jeltef.nl>,
	Thomas Munro <thomas.munro@gmail.com>, pgsql-hackers@postgresql.org,
	Zsolt Parragi <zsolt.parragi@percona.com>,
 Peter Eisentraut <peter@eisentraut.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Archived-At: 
 <https://www.postgresql.org/message-id/CAKZiRmwA6sMZqdumO1%2B_TmzyhqFAJ12Ufs-iRS1Zms%3DTndcYjQ%40mail.gmail.com>
Precedence: bulk

On Thu, May 28, 2026 at 9:42=E2=80=AFPM Andres Freund <andres@anarazel.de> =
wrote:
[..]
> It's definitely slower. I've not fully analyzed why, my suspicion is that=
 we
> end up being rather terribly IO bound - we used bigger and faster disks o=
n
> cirrus than we have access to with github hosted runners (there are large
> runners with more storage, but that's not free).
>
> A full testrun on master creates about 36GB of data directories. If indiv=
idual
> tests are fast, that's often not *that* bad, because the tests are over b=
efore
> linux decides to flush out the data, and then linux never needs to write =
that
> data back, because we remove the data directories immediately. But once y=
ou
> get to the point that several tests take more than 30s (the default time =
after
> which linux writes dirty data back) or enough dirty data accumulates (20%=
 of
> memory IIRC), you have a lot of IO.
>
> My buildfarm host, which hosts quite a few animals, got a new disk within=
 the
> last year. Here's what smartctl says about disk IO:
>
> Data Units Read:                    43,513,034 [22.2 TB]
> Data Units Written:                 6,062,401,949 [3.10 PB]
>
> A nice indication of how much our tests end up writing...

`nijna -C build test` (of course without compilation) that was run in
dedicated cgroup gave me this /sys/fs/cgroup/my_test_suite/io.stat
figure:
    rbytes=3D88616960 wbytes=3D18406756352 rios=3D13843 wios=3D275457 dbyte=
s=3D0 dios=3D0

~84.5 MB read
~17.1 GB written (sic!!!)
13k read ops
275k write ops

So yeah, I've really even didn't think we could generate that much IO there=
.
btw it's seems to be coming from block controller, so it's number of
flushed to disk (so the logically written data but removed without flush?
would be way higher; so by what Your' saying we should tweak that 30s
writeback right?) Anyway I've tried with way more relaxed dirty/writeback a=
nd
got this stil onthe same laptop with 32GB RAM:
    rbytes=3D20934656 wbytes=3D5957296128 rios=3D3040 wios=3D81992 dbytes=
=3D0 dios=3D

~20MB read
~5.55 GB written (down from 17GB)
3k read ops
82k write ops

And that was without even trying hard:
    sudo sysctl -w vm.dirty_ratio=3D50 # ~16GB
    sudo sysctl -w vm.dirty_background_ratio=3D40
    sudo sysctl -w vm.dirty_expire_centisecs=3D60000 #default was 3000 as Y=
ou said
    sudo sysctl -w vm.dirty_writeback_centisecs=3D50000

Steps were:
  sudo mkdir /sys/fs/cgroup/my_test_suite
  echo $$ | sudo tee /sys/fs/cgroup/my_test_suite/cgroup.procs
  ninja -C build test
  cat /sys/fs/cgroup/my_test_suite/io.stat

-J.