MIME-Version: 1.0
From: Koen De Groote <kdg.dev@gmail.com>
Date: Wed, 20 Nov 2024 12:16:00 +0100
Message-ID: <CAGbX52Fm=k8hHJKEzo6-mnh7gn91s=Lz_t6B5uF1SotpXH3UeA@mail.gmail.com>
Subject: Memory settings when running postgres in a docker container
To: PostgreSQL General <pgsql-general@lists.postgresql.org>
Content-Type: multipart/alternative; boundary="000000000000f4d63b06275646a3"
Archived-At: <https://www.postgresql.org/message-id/CAGbX52Fm%3Dk8hHJKEzo6-mnh7gn91s%3DLz_t6B5uF1SotpXH3UeA%40mail.gmail.com>
Precedence: bulk

--000000000000f4d63b06275646a3
Content-Type: text/plain; charset="UTF-8"

Assuming a machine with:

* 16 CPU cores
* 64GB RAM

Set to 500 max connections

A tool like this: https://pgtune.leopard.in.ua/

Will output recommended settings:

max_connections = 500
shared_buffers = 16GB
effective_cache_size = 48GB
maintenance_work_mem = 2GB
checkpoint_completion_target = 0.9
wal_buffers = 16MB
default_statistics_target = 100
random_page_cost = 1.1
effective_io_concurrency = 200
work_mem = 8388kB
huge_pages = try
min_wal_size = 1GB
max_wal_size = 4GB
max_worker_processes = 16
max_parallel_workers_per_gather = 4
max_parallel_workers = 16
max_parallel_maintenance_workers = 4

And they basically use up all the memory of the machine.

16GB shared buffers, 48GB effective cache size, 8MB of work_mem for some
reason...

This seems rather extreme. I feel there should be free memory for
emergencies and monitoring solutions.

And then there's the fact that postgres on this machine will be run in a
docker container. Which, on Linux, receives 64MB of /dev/shm shared memory
by default, but can be increased.

I feel like I should probably actually lower my upper limit for memory,
regardless of what the machine actually has, so I can have free memory, and
also not bring the container process itself into danger.

Is it as straightforward as putting my limit on, say 20GB, and then giving
more /dev/shm to the container? Or is there more to consider?

Regards,
Koen De Groote

--000000000000f4d63b06275646a3
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Assuming a machine with:<div><br></div><div>* 16 CPU cores=
</div><div>* 64GB RAM</div><div><br></div><div>Set to 500 max connections</=
div><div><br></div><div>A tool like this:=C2=A0<a href=3D"https://pgtune.le=
opard.in.ua/">https://pgtune.leopard.in.ua/</a></div><div><br></div><div>Wi=
ll output recommended settings:</div><div><br></div>max_connections =3D 500=
<br>shared_buffers =3D 16GB<br>effective_cache_size =3D 48GB<br>maintenance=
_work_mem =3D 2GB<br>checkpoint_completion_target =3D 0.9<br>wal_buffers =
=3D 16MB<br>default_statistics_target =3D 100<br>random_page_cost =3D 1.1<b=
r>effective_io_concurrency =3D 200<br>work_mem =3D 8388kB<br>huge_pages =3D=
 try<br>min_wal_size =3D 1GB<br>max_wal_size =3D 4GB<br>max_worker_processe=
s =3D 16<br>max_parallel_workers_per_gather =3D 4<br>max_parallel_workers =
=3D 16<br>max_parallel_maintenance_workers =3D 4<div><br></div><div>And the=
y basically use up all the memory of the machine.</div><div><br></div><div>=
16GB shared buffers, 48GB effective cache size, 8MB of work_mem for some re=
ason...</div><div><br></div><div>This seems rather extreme. I feel there sh=
ould be free memory for emergencies and monitoring solutions.</div><div><br=
></div><div>And then there&#39;s the fact that postgres on this machine wil=
l be run in a docker container. Which, on Linux, receives 64MB of /dev/shm =
shared memory by default, but can be increased.</div><div><br></div><div>I =
feel like I should probably actually lower my upper limit for memory, regar=
dless of what the machine actually has, so I can have free memory, and also=
 not bring the container process itself into danger.</div><div><br></div><d=
iv>Is it as straightforward as putting my limit on, say 20GB, and then givi=
ng more /dev/shm to the container? Or is there more to consider?</div><div>=
<br></div><div>Regards,</div><div>Koen De Groote</div><div><br></div><div><=
br></div><div><br></div><div><br></div><div><br></div><div><br></div></div>

--000000000000f4d63b06275646a3--