Date: Fri, 1 Nov 2024 20:56:17 +0100
From: "Peter J. Holzer" <hjp-pgsql@hjp.at>
To: pgsql-general@lists.postgresql.org
Subject: Re: Used memory calculation in containers - docker stats and file
 cache
Message-ID: <20241101195617.xkexgtwuyeo2hvmx@hjp.at>
Mail-Followup-To: pgsql-general@lists.postgresql.org
References: <CAJ+5Ff6qH6hLW3MVSip0eCY7pFdHJQvmFrOXfKRMQkqYVfsMHw@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha512;
	protocol="application/pgp-signature"; boundary="eqp2yoytrrxomnm3"
Content-Disposition: inline
In-Reply-To: <CAJ+5Ff6qH6hLW3MVSip0eCY7pFdHJQvmFrOXfKRMQkqYVfsMHw@mail.gmail.com>
Archived-At: <https://www.postgresql.org/message-id/20241101195617.xkexgtwuyeo2hvmx%40hjp.at>
Precedence: bulk


--eqp2yoytrrxomnm3
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On 2024-10-27 16:23:44 +0100, Costa Alexoglou wrote:
> The container limit was 16GB of RAM, and as soon as this limit was reache=
d,
> there was no restart or OOM errors, rather than a huge drop in memory (im=
age
> `ContainerRelativeAbsolute`).
[...]
> 2. What is happening on the OS level when suddenly 15GB of file cache is
> getting erased (image `ContainerRelativeAbsolute`)? I would expect for
> incremental deletes rather than so many GB of cache being evicted.

I don't know if Docker does anything strange here. I can think of two
scenarios which would normally result in a sudden drop in filesystem
cache size:

1) A large file (or many smaller files) which is cached is deleted
2) Something else briefly needs a lot of RAM, evicting data from the
   cache.

Both can happen in a database (for example, a large sort operation might
need a few GBs of either RAM or temporary files, depending on your
work_mem settings), but I wouldn't expect them to happen just before the
configured limit is reached. So I'd double check the logs if there are
any errors.

        hp

--=20
   _  | Peter J. Holzer    | Story must make more sense than reality.
|_|_) |                    |
| |   | hjp@hjp.at         |    -- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |       challenge!"

--eqp2yoytrrxomnm3
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEETtJbRjyPwVTYGJ5k8g5IURL+KF0FAmclMlcACgkQ8g5IURL+
KF0yuhAAmJ3YrEd0v51i1XoU45G4XLTLTh355rnom+ninmwUDtJmDw/c4Qd1+ZMU
lNi5iL1C9escXBIYwlMI6r25GN3LQTRmKPCVhxYecUumGjB2R4bKcHBZ36JKk16r
HgNWQxiB7ZSbyTghYpUjs+LS1/Tv+46UbUvp2FGSVleezDmbNm640r9mVUYLzMQ6
m5S9LNG6UbNUE6eJWWOSnvmRFz/WmB+YuMUWoBqwOhBFNeCvKyqMO85b55Ae2LK4
tyF+6Lcf19l+8BRYK4nREENgpwwcT9uVwJpateZ5ukSBCNlhtV9gt9zDkC4YAx2X
CF9MteqnaAariH5QJw55HD5S1HyMp5EdEND6aan6kizK5EUU4xF58PVV2tAu3VSr
Y2sxNREGiwogLywOVh/Hzpey/b0pAYvLMVs6DZT461oQP/k6nGyoapwmgwaxiwV9
UnSynp4IHJQRE4XV65kSOLsd3duxV/vFiClTB0a7vw87BJEO6YMaHYoSH3ca+u5m
ypkxZcc3xH6oHfKGlRVlm7dT4+UWkd0ur8M1IM0HWiBNkLbgR1ze8Zyz01W12Rux
bVR+BqYDOv1Q1ksbmMNWfmriDorTxp74hgOeOb9q+PSmsiIIv3aSNQIRs0tCX73D
YS4FfzT/incHFON93U5No4aK3psiGRQWov1Jz4qnz48yk68NFFI=
=FPs3
-----END PGP SIGNATURE-----

--eqp2yoytrrxomnm3--