pg.ddx.io pgsql-sql@postgresql.org mailing list archive
help / color / mirror / Atom feedRe: MacOS X Shared Buffers (SHMMAX)?
9+ messages / 5 participants
[nested] [flat]
* Re: MacOS X Shared Buffers (SHMMAX)?
@ 2002-05-21 01:59 Steve Lane <slane@fmpro.com>
parent: Command Prompt, Inc. <pgsql-general@commandprompt.com>
1 sibling, 1 reply; 9+ messages in thread
From: Steve Lane @ 2002-05-21 01:59 UTC (permalink / raw)
To: Command Prompt, Inc. <pgsql-general@commandprompt.com>; pgsql-general@postgresql.org; +Cc: jlx@commandprompt.com
On 5/20/02 9:25 PM, "Command Prompt, Inc." <pgsql-general@commandprompt.com>
wrote:
> Good day,
>
> Does anyone have any experience in increasing the amount of shared memory
> buffers available to Mac OS X/Darwin? When starting with more than 128
> backend connections on a test G3 I encounter:
>
> ------------------------------------------------------------------------------
> -----
> IpcMemoryCreate: shmget(key=5432001, size=33357824, 03600) failed: Invalid
> argument
>
> This error usually means that PostgreSQL's request for a shared memory
> segment exceeded your kernel's SHMMAX parameter. You can either
> reduce the request size or reconfigure the kernel with larger SHMMAX.
> ------------------------------------------------------------------------------
> -----
>
> However, I can find no helpful information looking around for how Darwin
> handles this configuration. Hopefully it does not require a recompile of
> the Kernel. ;)
Actually --- :-> I'm no expert on this topic, but Darwin is BSD and I do
recall a post on this very topic, to the effect that the poster believed
some flavors of BSD, possibly MOSX included, do require rebuilding the
kernel to change this value.
As a MOSX user I'd certainly love to be found wrong.
-- sgl
=======================================================
Steve Lane
Vice President
Chris Moyer Consulting, Inc.
833 West Chicago Ave Suite 203
Voice: (312) 433-2421 Email: slane@fmpro.com
Fax: (312) 850-3930 Web: http://www.fmpro.com
=======================================================
^ permalink raw reply [nested|flat] 9+ messages in thread
* MacOS X Shared Buffers (SHMMAX)?
@ 2002-05-21 02:25 Command Prompt, Inc. <pgsql-general@commandprompt.com>
0 siblings, 2 replies; 9+ messages in thread
From: Command Prompt, Inc. @ 2002-05-21 02:25 UTC (permalink / raw)
To: pgsql-general@postgresql.org; +Cc: jlx@commandprompt.com
Good day,
Does anyone have any experience in increasing the amount of shared memory
buffers available to Mac OS X/Darwin? When starting with more than 128
backend connections on a test G3 I encounter:
-----------------------------------------------------------------------------------
IpcMemoryCreate: shmget(key=5432001, size=33357824, 03600) failed: Invalid argument
This error usually means that PostgreSQL's request for a shared memory
segment exceeded your kernel's SHMMAX parameter. You can either
reduce the request size or reconfigure the kernel with larger SHMMAX.
-----------------------------------------------------------------------------------
However, I can find no helpful information looking around for how Darwin
handles this configuration. Hopefully it does not require a recompile of
the Kernel. ;)
Any knowledge on this topic would be greatly appreciated.
Best Regards,
Jw.
--
jlx@commandprompt.com
by way of pgsql-general@commandprompt.com
http://www.postgresql.info/
http://www.commandprompt.com/
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: MacOS X Shared Buffers (SHMMAX)?
@ 2002-05-21 03:09 Tom Lane <tgl@sss.pgh.pa.us>
parent: Command Prompt, Inc. <pgsql-general@commandprompt.com>
1 sibling, 0 replies; 9+ messages in thread
From: Tom Lane @ 2002-05-21 03:09 UTC (permalink / raw)
To: Command Prompt, Inc. <pgsql-general@commandprompt.com>; +Cc: pgsql-general@postgresql.org; jlx@commandprompt.com
"Command Prompt, Inc." <pgsql-general@commandprompt.com> writes:
> Does anyone have any experience in increasing the amount of shared memory
> buffers available to Mac OS X/Darwin?
A little bit of googling turns up Apple bug reports stating that OSX
has a hardwired limit on shared memory --- so no, short of modifying
and recompiling the kernel, you're outta luck.
regards, tom lane
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: MacOS X Shared Buffers (SHMMAX)?
@ 2002-05-22 17:02 Gregory Seidman <gss+pg@cs.brown.edu>
parent: Command Prompt, Inc. <pgsql-general@commandprompt.com>
1 sibling, 1 reply; 9+ messages in thread
From: Gregory Seidman @ 2002-05-22 17:02 UTC (permalink / raw)
To: pgsql-general@postgresql.org
Command Prompt, Inc. sez:
[...]
} [...actually, a minute later of looking at the source...]
}
} Okay, so it appears to be when it's checking sockets for data in
} src/backend/postmaster/pgstat.c, line 1579...I just tried re-building the
} kernel with a higher value for kern.ipc.maxsockets but that didn't have
} any effect, still get that error.
}
} Anyone have any ideas on what specifically would cause this part of pgstat
} to fail?
Recompiling the kernel was necessary to change the shared memory limit, but
kern.ipc.maxsockets can be changed on the fly with sysctl. It might be
getting changed in one of the scripts (perhaps SystemTuning) in
/System/Library/StartupItems (there are also sometimes scripts in
/Library/StartupItems; both directories are used in the startup process).
When your system is up and running, try sysctl kern.ipc.maxsockets and see
if the value is what you expected. If not, dig around in the StartupItems
directories and see if something is changing it.
} Best regards,
} Jw.
--Greg
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: MacOS X Shared Buffers (SHMMAX)?
@ 2002-05-22 17:20 Tom Lane <tgl@sss.pgh.pa.us>
parent: Command Prompt, Inc. <pgsql-general@commandprompt.com>
1 sibling, 1 reply; 9+ messages in thread
From: Tom Lane @ 2002-05-22 17:20 UTC (permalink / raw)
To: Command Prompt, Inc. <pgsql-general@commandprompt.com>; +Cc: Steve Lane <slane@fmpro.com>; jlx@commandprompt.com; pgsql-general@postgresql.org
"Command Prompt, Inc." <pgsql-general@commandprompt.com> writes:
> I was able to pull down the source for the Kernel, and increased the
> SHMMAX by a factor of 16. Upgraded to the new Kernel and I am now able to
> get safely 512 connections, even up around 900, but bumping it up to 1024,
> I run into the following error:
> PGSTATBUFF: recvfrom(2): Resource temporarily unavailable
> DEBUG: statistics collector process (pid 1988) exited with exit code 1
> ...which then repeats itself infinitely until the calling process is
> stopped. ;)
[ scratches head... ] "Resource temporarily unavailable" is EAGAIN
according to /usr/include/sys/errno.h on my OSX machine. But the man
page for recvfrom doesn't mention any plausible reasons for EAGAIN to
be signaled. select() just told us there was data available on the
socket, so WTF? Could this be a kernel bug?
You could try modifying pgstat.c to continue its loop rather than
exiting after it gets a recvfrom error. But if the error condition
recurs that'll just put pgstat.c into an infinite loop, so I'm not
sure this is any solution --- just a way of gathering more data.
regards, tom lane
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: MacOS X Shared Buffers (SHMMAX)?
@ 2002-05-22 17:30 Command Prompt, Inc. <pgsql-general@commandprompt.com>
parent: Steve Lane <slane@fmpro.com>
0 siblings, 2 replies; 9+ messages in thread
From: Command Prompt, Inc. @ 2002-05-22 17:30 UTC (permalink / raw)
To: Steve Lane <slane@fmpro.com>; +Cc: jlx@commandprompt.com; pgsql-general@postgresql.org
On Mon, 20 May 2002, Steve Lane wrote:
>Actually --- :-> I'm no expert on this topic, but Darwin is BSD and I do
>recall a post on this very topic, to the effect that the poster believed
>some flavors of BSD, possibly MOSX included, do require rebuilding the
>kernel to change this value.
>
>As a MOSX user I'd certainly love to be found wrong.
Well, after doing some digging in the OS X Darwin Developer archives, it
definitely appears that at this stage in Darwin's design it DOES require a
re-build of the Kernel -- thanks to those of you who pointed me in that
direction. ;)
Fortunately it's a fairly painless process I've found; you just need to
sign up with the Apple Open Source site, and you have instant CVS access.
I was able to pull down the source for the Kernel, and increased the
SHMMAX by a factor of 16. Upgraded to the new Kernel and I am now able to
get safely 512 connections, even up around 900, but bumping it up to 1024,
I run into the following error:
PGSTATBUFF: recvfrom(2): Resource temporarily unavailable
DEBUG: statistics collector process (pid 1988) exited with exit code 1
...which then repeats itself infinitely until the calling process is
stopped. ;)
I've upped the limit on the number of open files with ulimit, but I was
wondering if anyone knows what specifically invokes this "resource
temporarily unavailable" message from PGSTATBUFF? I don't see it in the
source casually grepping around, so is it I take it the result of the
recvfrom() function?
[...actually, a minute later of looking at the source...]
Okay, so it appears to be when it's checking sockets for data in
src/backend/postmaster/pgstat.c, line 1579...I just tried re-building the
kernel with a higher value for kern.ipc.maxsockets but that didn't have
any effect, still get that error.
Anyone have any ideas on what specifically would cause this part of pgstat
to fail?
Best regards,
Jw.
--
jlx@commandprompt.com
by way of pgsql-general@commandprompt.com
http://www.postgresql.info/
http://www.commandprompt.com/
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: MacOS X Shared Buffers (SHMMAX)?
@ 2002-05-22 18:57 Command Prompt, Inc. <pgsql-general@commandprompt.com>
parent: Gregory Seidman <gss+pg@cs.brown.edu>
0 siblings, 0 replies; 9+ messages in thread
From: Command Prompt, Inc. @ 2002-05-22 18:57 UTC (permalink / raw)
To: Gregory Seidman <gss+pg@cs.brown.edu>; +Cc: pgsql-general@postgresql.org
On Wed, 22 May 2002, Gregory Seidman wrote:
>Command Prompt, Inc. sez:
>[...]
>}Okay, so it appears to be when it's checking sockets for data in
>}src/backend/postmaster/pgstat.c, line 1579...I just tried re-building the
>}kernel with a higher value for kern.ipc.maxsockets but that didn't have
>}any effect, still get that error.
>}
>}Anyone have any ideas on what specifically would cause this part of pgstat
>}to fail?
>Recompiling the kernel was necessary to change the shared memory limit, but
>kern.ipc.maxsockets can be changed on the fly with sysctl.
Hm. I tried that, with "sysctl -w kern.ipcmaxsockets=2048", and got:
sysctl: oid 'kern.ipc.maxsockets' is read only
>It might be getting changed in one of the scripts (perhaps SystemTuning) in
>/System/Library/StartupItems (there are also sometimes scripts in
>/Library/StartupItems; both directories are used in the startup process).
>When your system is up and running, try sysctl kern.ipc.maxsockets and see
>if the value is what you expected. If not, dig around in the StartupItems
>directories and see if something is changing it.
After recompile it is actually the value I expected. It appears that the
problem is something less clear. :/
Jw.
--
jlx@commandprompt.com
by way of pgsql-general@commandprompt.com
http://www.postgresql.info/
http://www.commandprompt.com/
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: MacOS X Shared Buffers (SHMMAX)?
@ 2002-05-23 14:09 Nigel J. Andrews <nandrews@investsystems.co.uk>
parent: Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 9+ messages in thread
From: Nigel J. Andrews @ 2002-05-23 14:09 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Command Prompt, Inc. <pgsql-general@commandprompt.com>; Steve Lane <slane@fmpro.com>; jlx@commandprompt.com; pgsql-general@postgresql.org
On Wed, 22 May 2002, Tom Lane wrote:
> "Command Prompt, Inc." <pgsql-general@commandprompt.com> writes:
> > I was able to pull down the source for the Kernel, and increased the
> > SHMMAX by a factor of 16. Upgraded to the new Kernel and I am now able to
> > get safely 512 connections, even up around 900, but bumping it up to 1024,
> > I run into the following error:
>
> > PGSTATBUFF: recvfrom(2): Resource temporarily unavailable
> > DEBUG: statistics collector process (pid 1988) exited with exit code 1
>
> > ...which then repeats itself infinitely until the calling process is
> > stopped. ;)
>
> [ scratches head... ] "Resource temporarily unavailable" is EAGAIN
> according to /usr/include/sys/errno.h on my OSX machine. But the man
> page for recvfrom doesn't mention any plausible reasons for EAGAIN to
> be signaled. select() just told us there was data available on the
> socket, so WTF? Could this be a kernel bug?
>
> You could try modifying pgstat.c to continue its loop rather than
> exiting after it gets a recvfrom error. But if the error condition
> recurs that'll just put pgstat.c into an infinite loop, so I'm not
> sure this is any solution --- just a way of gathering more data.
My 'standard' way of doing this is to loop while the error is EAGAIN but with a
small retry counter to limit the number of attempts made. Hopefully the second
time through the call isn't interrupted again. Therefore I'm usually quite
happy saying that after something like 5 calls to a routine then there really
is an error.
Isn't this how people normally handle this?
--
Nigel J. Andrews
Director
---
Logictree Systems Limited
Computer Consultants
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: MacOS X Shared Buffers (SHMMAX)?
@ 2002-05-23 14:49 Tom Lane <tgl@sss.pgh.pa.us>
parent: Nigel J. Andrews <nandrews@investsystems.co.uk>
0 siblings, 0 replies; 9+ messages in thread
From: Tom Lane @ 2002-05-23 14:49 UTC (permalink / raw)
To: Nigel J. Andrews <nandrews@investsystems.co.uk>; +Cc: Command Prompt, Inc. <pgsql-general@commandprompt.com>; Steve Lane <slane@fmpro.com>; jlx@commandprompt.com; pgsql-general@postgresql.org
"Nigel J. Andrews" <nandrews@investsystems.co.uk> writes:
> My 'standard' way of doing this is to loop while the error is EAGAIN
> but with a small retry counter to limit the number of attempts
> made. Hopefully the second time through the call isn't interrupted
> again.
I think you're thinking of EINTR, not EAGAIN.
regards, tom lane
^ permalink raw reply [nested|flat] 9+ messages in thread
end of thread, other threads:[~2002-05-23 14:49 UTC | newest]
Thread overview: 9+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2002-05-21 02:25 MacOS X Shared Buffers (SHMMAX)? Command Prompt, Inc. <pgsql-general@commandprompt.com>
2002-05-21 01:59 ` Steve Lane <slane@fmpro.com>
2002-05-22 17:30 ` Command Prompt, Inc. <pgsql-general@commandprompt.com>
2002-05-22 17:02 ` Gregory Seidman <gss+pg@cs.brown.edu>
2002-05-22 18:57 ` Command Prompt, Inc. <pgsql-general@commandprompt.com>
2002-05-22 17:20 ` Tom Lane <tgl@sss.pgh.pa.us>
2002-05-23 14:09 ` Nigel J. Andrews <nandrews@investsystems.co.uk>
2002-05-23 14:49 ` Tom Lane <tgl@sss.pgh.pa.us>
2002-05-21 03:09 ` Tom Lane <tgl@sss.pgh.pa.us>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox