public inbox for [email protected]help / color / mirror / Atom feed
Running docker in postgres, SHM size of the docker container in postgres 16 4+ messages / 3 participants [nested] [flat]
* Running docker in postgres, SHM size of the docker container in postgres 16 @ 2024-11-19 22:22 Koen De Groote <[email protected]> 2024-11-19 23:37 ` Re: Running docker in postgres, SHM size of the docker container in postgres 16 Thomas Munro <[email protected]> 0 siblings, 1 reply; 4+ messages in thread From: Koen De Groote @ 2024-11-19 22:22 UTC (permalink / raw) To: PostgreSQL General <[email protected]> Hello all, Assume a machine with 16 CPU cores and 64GB of RAM. 300 max connections. Postgres is running here, inside a docker container. Docker containers receive a small amount of shared memory from /dev/shm Experience teaches that the default 64MB of /dev/shm for the postgres container, is not enough for databases of even 50GB in size. Previously I've increased the /dev/shm setting to 256MB for the postgres container. Issues that made this necessary all came in the form of too many queries running at once, asking too much memory space, I assume. I recently upgraded a setup from postgres 11 to postgres 16 via logical replication, and found that running "vacuum(analyze, verbose)" fails unless I give the container 1GB of /dev/shm. Where previously 256MB sufficed on the old postgres 11 version. Why would that be? It's the exact same data. The install is about 50GB in size. Is there something wrong with postgres 16, or did some settings significantly change, that I need to know about? I went over all the changelogs, nothing stood out as relevant to this, but that might be a lack of experience on my end. Regards, Koen De Groote ^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: Running docker in postgres, SHM size of the docker container in postgres 16 2024-11-19 22:22 Running docker in postgres, SHM size of the docker container in postgres 16 Koen De Groote <[email protected]> @ 2024-11-19 23:37 ` Thomas Munro <[email protected]> 2024-11-20 11:06 ` Re: Running docker in postgres, SHM size of the docker container in postgres 16 Koen De Groote <[email protected]> 0 siblings, 1 reply; 4+ messages in thread From: Thomas Munro @ 2024-11-19 23:37 UTC (permalink / raw) To: Koen De Groote <[email protected]>; +Cc: PostgreSQL General <[email protected]> On Wed, Nov 20, 2024 at 11:22 AM Koen De Groote <[email protected]> wrote: > Why would that be? It's the exact same data. The install is about 50GB in size. Is there something wrong with postgres 16, or did some settings significantly change, that I need to know about? I went over all the changelogs, nothing stood out as relevant to this, but that might be a lack of experience on my end. Parallel vacuum arrived in PostgreSQL 13, and that uses "dynamic shared memory" to share state between workers, and assuming dynamic_shared_memory_type=posix, that means shm_open(), which opens files under /dev/shm on Linux. ^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: Running docker in postgres, SHM size of the docker container in postgres 16 2024-11-19 22:22 Running docker in postgres, SHM size of the docker container in postgres 16 Koen De Groote <[email protected]> 2024-11-19 23:37 ` Re: Running docker in postgres, SHM size of the docker container in postgres 16 Thomas Munro <[email protected]> @ 2024-11-20 11:06 ` Koen De Groote <[email protected]> 2024-11-20 14:06 ` Re: Running docker in postgres, SHM size of the docker container in postgres 16 [email protected] 0 siblings, 1 reply; 4+ messages in thread From: Koen De Groote @ 2024-11-20 11:06 UTC (permalink / raw) To: Thomas Munro <[email protected]>; +Cc: PostgreSQL General <[email protected]> That explains a lot. I have the default of 2 max_parallel_maintenance_workers set, should I set this to 0? I realize this is of course an improvement, but working with docker containers, I'd like to avoid taking /dev/shm away from regular queries. I assume setting max_parallel_maintenance_workers to 0 is the fix here, is there perhaps something else I should know about, if I want to have control over this? Regards, Koen De Groote On Wed, Nov 20, 2024 at 12:38 AM Thomas Munro <[email protected]> wrote: > On Wed, Nov 20, 2024 at 11:22 AM Koen De Groote <[email protected]> wrote: > > Why would that be? It's the exact same data. The install is about 50GB > in size. Is there something wrong with postgres 16, or did some settings > significantly change, that I need to know about? I went over all the > changelogs, nothing stood out as relevant to this, but that might be a lack > of experience on my end. > > Parallel vacuum arrived in PostgreSQL 13, and that uses "dynamic > shared memory" to share state between workers, and assuming > dynamic_shared_memory_type=posix, that means shm_open(), which opens > files under /dev/shm on Linux. > ^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: Running docker in postgres, SHM size of the docker container in postgres 16 2024-11-19 22:22 Running docker in postgres, SHM size of the docker container in postgres 16 Koen De Groote <[email protected]> 2024-11-19 23:37 ` Re: Running docker in postgres, SHM size of the docker container in postgres 16 Thomas Munro <[email protected]> 2024-11-20 11:06 ` Re: Running docker in postgres, SHM size of the docker container in postgres 16 Koen De Groote <[email protected]> @ 2024-11-20 14:06 ` [email protected] 0 siblings, 0 replies; 4+ messages in thread From: [email protected] @ 2024-11-20 14:06 UTC (permalink / raw) To: [email protected] Docker databases do not perform well under heavy load. Everything can be configured to improve performance but going through the Docker disk access layer will significantly decrease performance. On Wednesday, November 20, 2024 at 06:06:36 AM GMT-5, Koen De Groote <[email protected]> wrote: That explains a lot. I have the default of 2 max_parallel_maintenance_workers set, should I set this to 0? I realize this is of course an improvement, but working with docker containers, I'd like to avoid taking /dev/shm away from regular queries. I assume setting max_parallel_maintenance_workers to 0 is the fix here, is there perhaps something else I should know about, if I want to have control over this? Regards,Koen De Groote On Wed, Nov 20, 2024 at 12:38 AM Thomas Munro <[email protected]> wrote: On Wed, Nov 20, 2024 at 11:22 AM Koen De Groote <[email protected]> wrote: > Why would that be? It's the exact same data. The install is about 50GB in size. Is there something wrong with postgres 16, or did some settings significantly change, that I need to know about? I went over all the changelogs, nothing stood out as relevant to this, but that might be a lack of experience on my end. Parallel vacuum arrived in PostgreSQL 13, and that uses "dynamic shared memory" to share state between workers, and assuming dynamic_shared_memory_type=posix, that means shm_open(), which opens files under /dev/shm on Linux. ^ permalink raw reply [nested|flat] 4+ messages in thread
end of thread, other threads:[~2024-11-20 14:06 UTC | newest] Thread overview: 4+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2024-11-19 22:22 Running docker in postgres, SHM size of the docker container in postgres 16 Koen De Groote <[email protected]> 2024-11-19 23:37 ` Thomas Munro <[email protected]> 2024-11-20 11:06 ` Koen De Groote <[email protected]> 2024-11-20 14:06 ` [email protected]
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox