public inbox for [email protected]  

help / color / mirror / Atom feed

From: Costa Alexoglou <[email protected]>
To: [email protected]
Subject: Used memory calculation in containers - docker stats and file cache
Date: Sun, 27 Oct 2024 16:23:44 +0100
Message-ID: <CAJ+5Ff6qH6hLW3MVSip0eCY7pFdHJQvmFrOXfKRMQkqYVfsMHw@mail.gmail.com> (raw)

Hey folks,

I noticed some behaviour which I was not expecting at all and after hours
of debugging I am getting somewhere, but still any help understanding what
is happening would be appreciated.

I was running a PostgreSQL server within a docker container, and after
starting to generate some moderate load (I/O heavy), I noticed the memory
usage was going up (reaching the container limit) and down, which reminded
me of a Heap GC event, but it was not.

The container limit was 16GB of RAM, and as soon as this limit was reached,
there was no restart or OOM errors, rather than a huge drop in memory
(image `ContainerRelativeAbsolute`).

After digging into this issue I found that this is file cache (image `Below
CLI`), and by default this measurement is included to docker stats CLI, and
is the default way of measuring in some observability tools as far as I can
tell.

I have two questions:
1. Should the file cache be indeed part of the reported memory that is
allocated to the container? The issue I see is that if we have an alerting
system, for example, on 90% of memory used in a container, this definitely
triggers many false positives. If we ignore the file cache, could this lead
to any reliability issues?
2. What is happening on the OS level when suddenly 15GB of file cache is
getting erased (image `ContainerRelativeAbsolute`)? I would expect for
incremental deletes rather than so many GB of cache being evicted.

Cheers,
Costa


Attachments:

  [image/png] BelowCLI.png (621.3K, 3-BelowCLI.png)
  download | view image

  [image/png] ContainerRelativeAbsolute.png (350.1K, 4-ContainerRelativeAbsolute.png)
  download | view image

view thread (4+ messages)  latest in thread

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: [email protected]
  Cc: [email protected], [email protected]
  Subject: Re: Used memory calculation in containers - docker stats and file cache
  In-Reply-To: <CAJ+5Ff6qH6hLW3MVSip0eCY7pFdHJQvmFrOXfKRMQkqYVfsMHw@mail.gmail.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox