Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1tEal3-008QgA-QO for pgsql-general@arkaria.postgresql.org; Fri, 22 Nov 2024 21:00:05 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1tEal1-00CN4H-R9 for pgsql-general@arkaria.postgresql.org; Fri, 22 Nov 2024 21:00:03 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1tEal1-00CN48-BQ for pgsql-general@lists.postgresql.org; Fri, 22 Nov 2024 21:00:03 +0000 Received: from mail-ej1-x629.google.com ([2a00:1450:4864:20::629]) by makus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1tEaky-003H59-6V for pgsql-general@lists.postgresql.org; Fri, 22 Nov 2024 21:00:01 +0000 Received: by mail-ej1-x629.google.com with SMTP id a640c23a62f3a-a9ec267b879so390602066b.2 for ; Fri, 22 Nov 2024 12:59:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1732309198; x=1732913998; darn=lists.postgresql.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=lxWk0dRskKOTuKuRFb7Rr747wpUOGrCZb9wHtEgZCrA=; b=fc9z213PC+9I8h6izL4QKfa4BpVBVdU02sEr2zYN/AT8HuC4hyKnnsxs+HNXXnYsU/ ci+HMmRLLkz7J8al+xcAyQZoLP3G8UfyypopxxbYyqGDQXfLV9CB1hxtlrOI9o2KiWFp v1z5dJyP5GlTXuKf16JEPT6nsJ12XEEm+yZNKRRDtHFkFbUR0dUMLIQMpW2e2Gqsjvp4 umSg9alUo1HD3oI21KVFHU31tpjniBhbw+xhXOaczECmLrCzQDc04D3QZEHS89EL9JzH bF4BQ2nGOCcDfhZSwU04sYpL1F7+IlUXPMxiyOuoX0tg/DDA2ixiHyp0b6j8052uVxpD KsNg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732309198; x=1732913998; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=lxWk0dRskKOTuKuRFb7Rr747wpUOGrCZb9wHtEgZCrA=; b=IwQykAWP26C7ZVX/r+JcXY2nVM8/Jr7e8g7FLpvPYFE8Ud4jdp0xHB9iyA43XepmY7 OpvkkL+KY6dOAkrfWl1dprcD75ErsAly7qCF8f7B4f1XueOvEoc4WS5WiA0owfneCNn+ R1H4iuyhATeScdTzDbMevRAMQsk/RVFggXQxbXP9UJlYx3igs2c/KgaqeV6TntTJCppB pA6zb9xA6hQ/wLay26H19Ww7IFRTUtxSDzt6NnLxgpphd9rNP75J/CIZp2J1o0r/gxbR SuIZuvfrb10EyTUljDZNQH9YeVB8CItp5LnEXFbgbn58cYVCCXmdf7nycwiLS3Pysnah or/g== X-Gm-Message-State: AOJu0YyXk0sQEZvmR8AwtPfGHnjqEt15E9KLik+cUfUKqH/F0CdbthhY 6774tFy/0GEpNkGU7Zsg2HZvY/PRnb4pQu9kdd6Q+dF4fiAevsV7cKMmIgGuBLaN+ez7LoXjIHu JNSsSvNC2gDJl62HQdBbZdKVgEYI= X-Gm-Gg: ASbGncuVON2e7d9Zh+Q2mpoezmXx/rEXdI/g1e6Pt9ZWnAlraUybghAJJRQ+oHDUctj HRUiFVbHBjxrKtt5JXp2W1ZS6rLnVUpVkHhhNZaEFLuvgXWV3IqdzGWHshh1zPBv6dQ== X-Google-Smtp-Source: AGHT+IFvuUdELZrsOiYGYbVrA3wuenAnsOad1eZk1nGofPT5EsLSrfELBFRaqbnq8G1UI8SPOgOVMIoHV7H43m842p8= X-Received: by 2002:a17:907:7741:b0:aa5:324f:5319 with SMTP id a640c23a62f3a-aa5324f537dmr12945266b.12.1732309197772; Fri, 22 Nov 2024 12:59:57 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Koen De Groote Date: Fri, 22 Nov 2024 21:59:46 +0100 Message-ID: Subject: Re: Memory settings when running postgres in a docker container To: David Mullineux Cc: PostgreSQL General Content-Type: multipart/alternative; boundary="0000000000005fe4e3062786aa02" List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --0000000000005fe4e3062786aa02 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Ah, see, I didn't know that. On Wed, Nov 20, 2024 at 11:10=E2=80=AFPM David Mullineux = wrote: > i dont get why you think all memroy will be used. > When you say > shared_buffers =3D 16GB > effective_cache_size =3D 48GB > > ...then this is using only 16GB for shared buffers. > > The effective _cache_size doesn't cause any memory to.be allocated. It's > just a hint to optomizer .... > > On Wed, 20 Nov 2024, 11:16 Koen De Groote, wrote: > >> Assuming a machine with: >> >> * 16 CPU cores >> * 64GB RAM >> >> Set to 500 max connections >> >> A tool like this: https://pgtune.leopard.in.ua/ >> >> Will output recommended settings: >> >> max_connections =3D 500 >> shared_buffers =3D 16GB >> effective_cache_size =3D 48GB >> maintenance_work_mem =3D 2GB >> checkpoint_completion_target =3D 0.9 >> wal_buffers =3D 16MB >> default_statistics_target =3D 100 >> random_page_cost =3D 1.1 >> effective_io_concurrency =3D 200 >> work_mem =3D 8388kB >> huge_pages =3D try >> min_wal_size =3D 1GB >> max_wal_size =3D 4GB >> max_worker_processes =3D 16 >> max_parallel_workers_per_gather =3D 4 >> max_parallel_workers =3D 16 >> max_parallel_maintenance_workers =3D 4 >> >> And they basically use up all the memory of the machine. >> >> 16GB shared buffers, 48GB effective cache size, 8MB of work_mem for some >> reason... >> >> This seems rather extreme. I feel there should be free memory for >> emergencies and monitoring solutions. >> >> And then there's the fact that postgres on this machine will be run in a >> docker container. Which, on Linux, receives 64MB of /dev/shm shared memo= ry >> by default, but can be increased. >> >> I feel like I should probably actually lower my upper limit for memory, >> regardless of what the machine actually has, so I can have free memory, = and >> also not bring the container process itself into danger. >> >> Is it as straightforward as putting my limit on, say 20GB, and then >> giving more /dev/shm to the container? Or is there more to consider? >> >> Regards, >> Koen De Groote >> >> >> >> >> >> >> --0000000000005fe4e3062786aa02 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Ah, see, I didn't know that.

On Wed, Nov 20, 2024= at 11:10=E2=80=AFPM David Mullineux <dmullx@gmail.com> wrote:
i dont get why you think all memroy wil= l be used.
=C2=A0When you say
shared_buffers =3D 16GB
effective_cache_size =3D 48G= B

...then this is u= sing only 16GB for shared buffers.

The effective _cache_size doesn't cause any memory to.be allocated. It's just a= hint to optomizer ....

On Wed, 20 Nov 2024, 11:16 Koen De Groo= te, <kdg.dev@gmai= l.com> wrote:
Assuming a machine with:

* 16 CPU = cores
* 64GB RAM

Set to 500 max connecti= ons

A tool like this:=C2=A0https://pgtune.leop= ard.in.ua/

Will output recommended settings:

max_connections =3D 500
shared_buffers =3D 16GB
ef= fective_cache_size =3D 48GB
maintenance_work_mem =3D 2GB
checkpoint_c= ompletion_target =3D 0.9
wal_buffers =3D 16MB
default_statistics_targ= et =3D 100
random_page_cost =3D 1.1
effective_io_concurrency =3D 200<= br>work_mem =3D 8388kB
huge_pages =3D try
min_wal_size =3D 1GB
max= _wal_size =3D 4GB
max_worker_processes =3D 16
max_parallel_workers_pe= r_gather =3D 4
max_parallel_workers =3D 16
max_parallel_maintenance_w= orkers =3D 4

And they basically use up all the memory of= the machine.

16GB shared buffers, 48GB effective = cache size, 8MB of work_mem for some reason...

Thi= s seems rather extreme. I feel there should be free memory for emergencies = and monitoring solutions.

And then there's the= fact that postgres on this machine will be run in a docker container. Whic= h, on Linux, receives 64MB of /dev/shm shared memory by default, but can be= increased.

I feel like I should probably actually= lower my upper limit for memory, regardless of what the machine actually h= as, so I can have free memory, and also not bring the container process its= elf into danger.

Is it as straightforward as putti= ng my limit on, say 20GB, and then giving more /dev/shm to the container? O= r is there more to consider?

Regards,
Ko= en De Groote






--0000000000005fe4e3062786aa02--