pg.ddx.io pgsql-admin@postgresql.org mailing list archive
help / color / mirror / Atom feedshm
7+ messages / 3 participants
[nested] [flat]
* shm
@ 2020-03-05 14:43 Scott Ribe <scott_ribe@elevated-dev.com>
0 siblings, 2 replies; 7+ messages in thread
From: Scott Ribe @ 2020-03-05 14:43 UTC (permalink / raw)
To: pgsql-admin
What is POSIX shared memory used for anymore? Shared buffers now defaults to mmap, right? So when I run out of shm in Kubernetes because of the 64MB limit, what is actually causing it to run out?
(I know about creating the memory-based volume and mounting it at /dev/shm, and have implemented that workaround. Still want to know what is causing this memory use.)
--
Scott Ribe
scott_ribe@elevated-dev.com
https://www.linkedin.com/in/scottribe/
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: shm
@ 2020-03-05 14:51 Tom Lane <tgl@sss.pgh.pa.us>
parent: Scott Ribe <scott_ribe@elevated-dev.com>
1 sibling, 2 replies; 7+ messages in thread
From: Tom Lane @ 2020-03-05 14:51 UTC (permalink / raw)
To: Scott Ribe <scott_ribe@elevated-dev.com>; +Cc: pgsql-admin
Scott Ribe <scott_ribe@elevated-dev.com> writes:
> What is POSIX shared memory used for anymore?
We still allocate a small shm block as a means of counting live backends
(since the SysV spec includes an "nattch" count but mmap'd segments do
not provide that info). Should only be ~50 bytes though.
regards, tom lane
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: shm
@ 2020-03-05 16:10 Imre Samu <pella.samu@gmail.com>
parent: Scott Ribe <scott_ribe@elevated-dev.com>
1 sibling, 0 replies; 7+ messages in thread
From: Imre Samu @ 2020-03-05 16:10 UTC (permalink / raw)
To: Scott Ribe <scott_ribe@elevated-dev.com>; +Cc: pgsql-admin
> So when I run out of shm in Kubernetes because of the 64MB limit, what is
actually causing it to run out?
on (postgres) docker it is a frequent problem and documented .. and the
result is a simple ERROR ... :(
and in: https://github.com/docker-library/docs/tree/master/postgres
*"Also note that the default /dev/shm size for containers is 64MB. If the
shared memory is exhausted you will encounter ERROR: could not resize
shared memory segment . . . : No space left on device. You will want to
pass --shm-size=256MB
<https://docs.docker.com/engine/reference/run/#runtime-constraints-on-resources;
for
example to docker run, or alternatively in docker-compose
<https://docs.docker.com/compose/compose-file/#domainname-hostname-ipc-mac_address-privileged-read_on...
https://github.com/docker-library/postgres/issues/416
https://stackoverflow.com/questions/56839975/docker-shm-size-dev-shm-resizing-shared-memory
best,
Imre
Scott Ribe <scott_ribe@elevated-dev.com> ezt írta (időpont: 2020. márc. 5.,
Cs, 15:43):
> What is POSIX shared memory used for anymore? Shared buffers now defaults
> to mmap, right? So when I run out of shm in Kubernetes because of the 64MB
> limit, what is actually causing it to run out?
>
> (I know about creating the memory-based volume and mounting it at
> /dev/shm, and have implemented that workaround. Still want to know what is
> causing this memory use.)
>
> --
> Scott Ribe
> scott_ribe@elevated-dev.com
> https://www.linkedin.com/in/scottribe/
>
>
>
>
>
>
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: shm
@ 2020-03-06 14:03 Scott Ribe <scott_ribe@elevated-dev.com>
parent: Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 0 replies; 7+ messages in thread
From: Scott Ribe @ 2020-03-06 14:03 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: pgsql-admin
> On Mar 5, 2020, at 7:51 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
> Scott Ribe <scott_ribe@elevated-dev.com> writes:
>> What is POSIX shared memory used for anymore?
>
> We still allocate a small shm block as a means of counting live backends
> (since the SysV spec includes an "nattch" count but mmap'd segments do
> not provide that info). Should only be ~50 bytes though.
>
> regards, tom lane
I recreated the error so I could post it here:
ERROR 53100 (disk_full) could not resize shared memory segment "/PostgreSQL.935653392" to 1048576 bytes: No space left on device
So something is using shared memory, and wants 1MB chunks. What could that be?
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: shm
@ 2020-03-06 14:06 Scott Ribe <scott_ribe@elevated-dev.com>
parent: Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 1 reply; 7+ messages in thread
From: Scott Ribe @ 2020-03-06 14:06 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: pgsql-admin
> On Mar 5, 2020, at 7:51 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
> We still allocate a small shm block as a means of counting live backends
> (since the SysV spec includes an "nattch" count but mmap'd segments do
> not provide that info). Should only be ~50 bytes though.
I recreated the error so I could post it here:
ERROR 53100 (disk_full) could not resize shared memory segment "/PostgreSQL.935653392" to 1048576 bytes: No space left on device
So something is using shared memory, and wants 1MB chunks. What could that be?
This is on PG 11.7
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: shm
@ 2020-03-06 15:34 Tom Lane <tgl@sss.pgh.pa.us>
parent: Scott Ribe <scott_ribe@elevated-dev.com>
0 siblings, 1 reply; 7+ messages in thread
From: Tom Lane @ 2020-03-06 15:34 UTC (permalink / raw)
To: Scott Ribe <scott_ribe@elevated-dev.com>; +Cc: pgsql-admin
Scott Ribe <scott_ribe@elevated-dev.com> writes:
>> On Mar 5, 2020, at 7:51 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> We still allocate a small shm block as a means of counting live backends
>> (since the SysV spec includes an "nattch" count but mmap'd segments do
>> not provide that info). Should only be ~50 bytes though.
> I recreated the error so I could post it here:
> ERROR 53100 (disk_full) could not resize shared memory segment "/PostgreSQL.935653392" to 1048576 bytes: No space left on device
Oh, if it's trying to resize then this is dynamic shared memory, not
the small static block. On most platforms we use POSIX shm_open() for
that purpose, not SysV-style shm ... what have you got
dynamic_shared_memory_type set to?
It could be that the space limit applies to both APIs anyway, in which
case your choices are to raise the limit or disable parallel query.
regards, tom lane
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: shm
@ 2020-03-06 16:32 Scott Ribe <scott_ribe@elevated-dev.com>
parent: Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 0 replies; 7+ messages in thread
From: Scott Ribe @ 2020-03-06 16:32 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: pgsql-admin
> On Mar 6, 2020, at 8:34 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
> Oh, if it's trying to resize then this is dynamic shared memory, not
> the small static block. On most platforms we use POSIX shm_open() for
> that purpose, not SysV-style shm ... what have you got
> dynamic_shared_memory_type set to?
I just confirmed that it is POSIX. (I'm pretty sure we've never even considered changing the default.) So it would seem that, in this context at least, the limit applies to it. And the context is containerized, Kubernetes, where there is no option to set the container's shm limit so we just have to use the shm mount trick.
> It could be that the space limit applies to both APIs anyway, in which
> case your choices are to raise the limit or disable parallel query.
Well if it's coming from parallel query then I'm OK with the workaround. Primarily I wanted to make sure it wasn't caused by something dumb/worthless that we should just not do. (For instance, for historical reasons, we are temporarily running with max_connections way higher than is good practice...)
^ permalink raw reply [nested|flat] 7+ messages in thread
end of thread, other threads:[~2020-03-06 16:32 UTC | newest]
Thread overview: 7+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2020-03-05 14:43 shm Scott Ribe <scott_ribe@elevated-dev.com>
2020-03-05 14:51 ` Tom Lane <tgl@sss.pgh.pa.us>
2020-03-06 14:03 ` Scott Ribe <scott_ribe@elevated-dev.com>
2020-03-06 14:06 ` Scott Ribe <scott_ribe@elevated-dev.com>
2020-03-06 15:34 ` Tom Lane <tgl@sss.pgh.pa.us>
2020-03-06 16:32 ` Scott Ribe <scott_ribe@elevated-dev.com>
2020-03-05 16:10 ` Imre Samu <pella.samu@gmail.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox