pg.ddx.io pgsql-performance@postgresql.org mailing list archive
help / color / mirror / Atom feedLinux memory zone reclaim
19+ messages / 8 participants
[nested] [flat]
* Linux memory zone reclaim
@ 2012-07-18 01:52 Greg Smith <greg@2ndQuadrant.com>
2012-07-18 02:00 ` Re: Linux memory zone reclaim Scott Marlowe <scott.marlowe@gmail.com>
2012-07-24 18:36 ` Re: Linux memory zone reclaim John Lister <john.lister@kickstone.com>
2012-07-27 00:25 ` Re: Linux memory zone reclaim Bruce Momjian <bruce@momjian.us>
2012-07-30 16:43 ` Re: Linux memory zone reclaim Kevin Grittner <Kevin.Grittner@wicourts.gov>
0 siblings, 4 replies; 19+ messages in thread
From: Greg Smith @ 2012-07-18 01:52 UTC (permalink / raw)
To: pgsql-performance
Newer Linux systems with lots of cores have a problem I've been running
into a lot more lately I wanted to share initial notes on. My "newer"
means running the 2.6.32 kernel or later, since I mostly track
"enterprise" Linux distributions like RHEL6 and Debian Squeeze. The
issue is around Linux's zone_reclaim feature. When it pops up, turning
that feature off help a lot. Details on what I understand of the
problem are below, and as always things may have changed already in even
newer kernels.
zone_reclaim tries to optimize memory speed on NUMA systems with more
than one CPU socket. There some banks of memory that can be "closer" to
a particular socket, as measured by transfer rate, because of how the
memory is routed to the various cores on each socket. There is no true
default for this setting. Linux checks the hardware and turns this
on/off based on what transfer rate it sees between NUMA nodes, where
there are more than one and its test shows some distance between them.
You can tell if this is turned on like this:
echo /proc/sys/vm/zone_reclaim_mode
Where 1 means it's enabled. Install the numactl utility and you can see
why it's made that decision:
# numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 12 13 14 15 16 17
node 0 size: 73718 MB
node 0 free: 419 MB
node 1 cpus: 6 7 8 9 10 11 18 19 20 21 22 23
node 1 size: 73728 MB
node 1 free: 30 MB
node distances:
node 0 1
0: 10 21
1: 21 10
Note how the "distance" for a transfer from node 0->0 or 1->1 is 10
units, while 0->1 or 1->0 is 21. That what's tested at boot time, where
the benchmarked speed is turned into this abstract distance number. And
if there is a large difference in cross-zone timing, then zone reclaim
is enabled.
Scott Marlowe has been griping about this on the mailing lists here for
a while now, and it's increasingly trouble for systems I've been seeing
lately too. This is a well known problem with MySQL:
http://blog.jcole.us/2010/09/28/mysql-swap-insanity-and-the-numa-architecture/
and NUMA issues have impacted Oracle too. On PostgreSQL shared_buffers
isn't normally set as high as MySQL's buffer cache, making it a bit less
vulnerable to this class of problem. But it's surely still a big
problem for PostgreSQL on some systems.
I've taken to disabling /proc/sys/vm/zone_reclaim_mode on any Linux
system where it's turned on now. I'm still working through whether it
also makes sense in all cases to use the more complicated memory
interleaving suggestions that MySQL users have implemented, something
most people would need to push into their PostgreSQL server started up
scripts in /etc/init.d (That will be a fun rpm/deb packaging issue to
deal with if this becomes more wide-spread) Suggestions on whether that
is necessary, or if just disabling zone_reclaim is enough, are welcome
from anyone who wants to try and benchmark it.
Note that this is all tricky to test because some of the bad behavior
only happens when the server runs this zone reclaim method, which isn't
a trivial situation to create at will. Servers that have this problem
tend to have it pop up intermittently, you'll see one incredibly slow
query periodically while most are fast. All depends on exactly what
core is executing, where the memory it needs is at, and whether the
server wants to reclaim memory (and just what that means its own
complicated topic) as part of that.
--
Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: Linux memory zone reclaim
2012-07-18 01:52 Linux memory zone reclaim Greg Smith <greg@2ndQuadrant.com>
@ 2012-07-18 02:00 ` Scott Marlowe <scott.marlowe@gmail.com>
2012-07-18 03:51 ` Re: Linux memory zone reclaim Dave Crooke <dcrooke@gmail.com>
2012-07-18 05:38 ` Re: Linux memory zone reclaim Claudio Freire <klaussfreire@gmail.com>
3 siblings, 2 replies; 19+ messages in thread
From: Scott Marlowe @ 2012-07-18 02:00 UTC (permalink / raw)
To: Greg Smith <greg@2ndquadrant.com>; +Cc: pgsql-performance
On Tue, Jul 17, 2012 at 7:52 PM, Greg Smith <greg@2ndquadrant.com> wrote:
> Newer Linux systems with lots of cores have a problem I've been running into
> a lot more lately I wanted to share initial notes on. My "newer" means
> running the 2.6.32 kernel or later, since I mostly track "enterprise" Linux
> distributions like RHEL6 and Debian Squeeze. The issue is around Linux's
> zone_reclaim feature. When it pops up, turning that feature off help a lot.
> Details on what I understand of the problem are below, and as always things
> may have changed already in even newer kernels.
SNIP
> Scott Marlowe has been griping about this on the mailing lists here for a
> while now, and it's increasingly trouble for systems I've been seeing lately
> too. This is a well known problem with MySQL:
> http://blog.jcole.us/2010/09/28/mysql-swap-insanity-and-the-numa-architecture/
Thanks for the link, I'll read up on it. I do have access to large
(24 to 40 core) NUMA machines so I might try some benchmarking on them
to see how they work.
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: Linux memory zone reclaim
2012-07-18 01:52 Linux memory zone reclaim Greg Smith <greg@2ndQuadrant.com>
2012-07-18 02:00 ` Re: Linux memory zone reclaim Scott Marlowe <scott.marlowe@gmail.com>
@ 2012-07-18 03:51 ` Dave Crooke <dcrooke@gmail.com>
1 sibling, 0 replies; 19+ messages in thread
From: Dave Crooke @ 2012-07-18 03:51 UTC (permalink / raw)
To: Scott Marlowe <scott.marlowe@gmail.com>; +Cc: pgsql-performance@postgresql.org, Greg Smith <greg@2ndquadrant.com>
On the larger, cellular Itanium systems with multiple motherboards (rx6600
to Superdome) Oracle has done a lot of tuning with the HP-UX kernel calls
to optimize for NUMA issues. Will be interesting to see what they bring to
Linux.
On Jul 17, 2012 9:01 PM, "Scott Marlowe" <scott.marlowe@gmail.com> wrote:
> On Tue, Jul 17, 2012 at 7:52 PM, Greg Smith <greg@2ndquadrant.com> wrote:
> > Newer Linux systems with lots of cores have a problem I've been running
> into
> > a lot more lately I wanted to share initial notes on. My "newer" means
> > running the 2.6.32 kernel or later, since I mostly track "enterprise"
> Linux
> > distributions like RHEL6 and Debian Squeeze. The issue is around Linux's
> > zone_reclaim feature. When it pops up, turning that feature off help a
> lot.
> > Details on what I understand of the problem are below, and as always
> things
> > may have changed already in even newer kernels.
>
> SNIP
>
> > Scott Marlowe has been griping about this on the mailing lists here for a
> > while now, and it's increasingly trouble for systems I've been seeing
> lately
> > too. This is a well known problem with MySQL:
> >
> http://blog.jcole.us/2010/09/28/mysql-swap-insanity-and-the-numa-architecture/
>
> Thanks for the link, I'll read up on it. I do have access to large
> (24 to 40 core) NUMA machines so I might try some benchmarking on them
> to see how they work.
>
> --
> Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-performance
>
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: Linux memory zone reclaim
2012-07-18 01:52 Linux memory zone reclaim Greg Smith <greg@2ndQuadrant.com>
2012-07-18 02:00 ` Re: Linux memory zone reclaim Scott Marlowe <scott.marlowe@gmail.com>
@ 2012-07-18 05:38 ` Claudio Freire <klaussfreire@gmail.com>
1 sibling, 0 replies; 19+ messages in thread
From: Claudio Freire @ 2012-07-18 05:38 UTC (permalink / raw)
To: Scott Marlowe <scott.marlowe@gmail.com>; +Cc: Greg Smith <greg@2ndquadrant.com>; pgsql-performance
On Tue, Jul 17, 2012 at 11:00 PM, Scott Marlowe <scott.marlowe@gmail.com> wrote:
>
> Thanks for the link, I'll read up on it. I do have access to large
> (24 to 40 core) NUMA machines so I might try some benchmarking on them
> to see how they work.
It must have been said already, but I'll repeat it just in case:
I think postgres has an easy solution. Spawn the postmaster with
"interleave", to allocate shared memory, and then switch to "local" on
the backends.
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: Linux memory zone reclaim
2012-07-18 01:52 Linux memory zone reclaim Greg Smith <greg@2ndQuadrant.com>
@ 2012-07-24 18:36 ` John Lister <john.lister@kickstone.com>
2012-07-24 18:41 ` Re: Linux memory zone reclaim Claudio Freire <klaussfreire@gmail.com>
3 siblings, 1 reply; 19+ messages in thread
From: John Lister @ 2012-07-24 18:36 UTC (permalink / raw)
To: pgsql-performance
On Tue, Jul 18, 2012 at 2:38 AM, Claudio Freire wrote:
>It must have been said already, but I'll repeat it just in case:
>I think postgres has an easy solution. Spawn the postmaster with
>"interleave", to allocate shared memory, and then switch to "local" on
>the backends.
Do you have a suggestion about how to do that? I'm running Ubuntu 12.04
and PG 9.1, I've modified pg_ctlcluster to cause pg_ctl to use a wrapper
script which starts the postmaster using a numactl wrapper, but all
subsequent client processes are started with interleaving enabled as
well. Any ideas how to make just the postmaster process start with
interleaving?
Thanks
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: Linux memory zone reclaim
2012-07-18 01:52 Linux memory zone reclaim Greg Smith <greg@2ndQuadrant.com>
2012-07-24 18:36 ` Re: Linux memory zone reclaim John Lister <john.lister@kickstone.com>
@ 2012-07-24 18:41 ` Claudio Freire <klaussfreire@gmail.com>
2012-07-24 20:12 ` Re: Linux memory zone reclaim Claudio Freire <klaussfreire@gmail.com>
0 siblings, 1 reply; 19+ messages in thread
From: Claudio Freire @ 2012-07-24 18:41 UTC (permalink / raw)
To: John Lister <john.lister@kickstone.com>; +Cc: pgsql-performance
On Tue, Jul 24, 2012 at 3:36 PM, John Lister <john.lister@kickstone.com> wrote:
> Do you have a suggestion about how to do that? I'm running Ubuntu 12.04 and
> PG 9.1, I've modified pg_ctlcluster to cause pg_ctl to use a wrapper script
> which starts the postmaster using a numactl wrapper, but all subsequent
> client processes are started with interleaving enabled as well. Any ideas
> how to make just the postmaster process start with interleaving?
postmaster should call numactl right after forking:
http://linux.die.net/man/2/set_mempolicy
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: Linux memory zone reclaim
2012-07-18 01:52 Linux memory zone reclaim Greg Smith <greg@2ndQuadrant.com>
2012-07-24 18:36 ` Re: Linux memory zone reclaim John Lister <john.lister@kickstone.com>
2012-07-24 18:41 ` Re: Linux memory zone reclaim Claudio Freire <klaussfreire@gmail.com>
@ 2012-07-24 20:12 ` Claudio Freire <klaussfreire@gmail.com>
2012-07-24 21:00 ` Re: Linux memory zone reclaim Claudio Freire <klaussfreire@gmail.com>
2012-07-24 21:23 ` Re: Linux memory zone reclaim John Lister <john.lister@kickstone.com>
0 siblings, 2 replies; 19+ messages in thread
From: Claudio Freire @ 2012-07-24 20:12 UTC (permalink / raw)
To: John Lister <john.lister@kickstone.com>; +Cc: pgsql-performance
On Tue, Jul 24, 2012 at 3:41 PM, Claudio Freire <klaussfreire@gmail.com> wrote:
> On Tue, Jul 24, 2012 at 3:36 PM, John Lister <john.lister@kickstone.com> wrote:
>> Do you have a suggestion about how to do that? I'm running Ubuntu 12.04 and
>> PG 9.1, I've modified pg_ctlcluster to cause pg_ctl to use a wrapper script
>> which starts the postmaster using a numactl wrapper, but all subsequent
>> client processes are started with interleaving enabled as well. Any ideas
>> how to make just the postmaster process start with interleaving?
>
> postmaster should call numactl right after forking:
> http://linux.die.net/man/2/set_mempolicy
Something like the attached patch (untested)
Attachments:
numa.patch (application/octet-stream, 2.5K) download
diff --git a/configure.in b/configure.in
index 8ed09f5..44eff38 100644
--- a/configure.in
+++ b/configure.in
@@ -19,10 +19,10 @@ m4_pattern_forbid(^PGAC_)dnl to catch undefined macros
AC_INIT([PostgreSQL], [9.3devel], [pgsql-bugs@postgresql.org])
-m4_if(m4_defn([m4_PACKAGE_VERSION]), [2.63], [], [m4_fatal([Autoconf version 2.63 is required.
-Untested combinations of 'autoconf' and PostgreSQL versions are not
-recommended. You can remove the check from 'configure.in' but it is then
-your responsibility whether the result works or not.])])
+#m4_if(m4_defn([m4_PACKAGE_VERSION]), [2.63], [], [m4_fatal([Autoconf version 2.63 is required.
+#Untested combinations of 'autoconf' and PostgreSQL versions are not
+#recommended. You can remove the check from 'configure.in' but it is then
+#your responsibility whether the result works or not.])])
AC_COPYRIGHT([Copyright (c) 1996-2012, PostgreSQL Global Development Group])
AC_CONFIG_SRCDIR([src/backend/access/common/heaptuple.c])
AC_CONFIG_AUX_DIR(config)
@@ -604,6 +604,17 @@ PGAC_ARG_BOOL(with, gssapi, no, [build with GSSAPI support],
AC_MSG_RESULT([$with_gssapi])
#
+# NUMA
+#
+AC_MSG_CHECKING([whether to build with NUMA support])
+PGAC_ARG_BOOL(with, numa, no, [build with NUMA support],
+[
+ AC_DEFINE(ENABLE_NUMA, 1, [Define to build with NUMA support. (--with-numa)])
+])
+AC_MSG_RESULT([$with_numa])
+AC_SUBST(with_numa)
+
+#
# Kerberos 5
#
AC_MSG_CHECKING([whether to build with Kerberos 5 support])
@@ -922,6 +933,15 @@ if test "$with_gssapi" = yes ; then
fi
fi
+if test "$with_numa" = yes ; then
+ if test "$PORTNAME" != "win32"; then
+ AC_SEARCH_LIBS(set_mempolicy, [numa], [],
+ [AC_MSG_ERROR([could not find function 'set_mempolicy' required for NUMA])])
+ else
+ LIBS="$LIBS -lnuma"
+ fi
+fi
+
if test "$with_krb5" = yes ; then
if test "$PORTNAME" != "win32"; then
AC_SEARCH_LIBS(com_err, [krb5 'krb5 -lcrypto -ldes -lasn1 -lroken' com_err 'com_err -lssl -lcrypto'], [],
diff --git a/src/backend/postmaster/fork_process.c b/src/backend/postmaster/fork_process.c
index 2a415e8..7997502 100644
--- a/src/backend/postmaster/fork_process.c
+++ b/src/backend/postmaster/fork_process.c
@@ -18,6 +18,10 @@
#include <sys/time.h>
#include <unistd.h>
+#ifdef ENABLE_NUMA
+#include <numaif.h>
+#endif
+
#ifndef WIN32
/*
* Wrapper for fork(). Return values are the same as those for fork():
@@ -124,6 +128,10 @@ fork_process(void)
}
}
#endif /* LINUX_OOM_ADJ */
+
+#ifdef ENABLE_NUMA
+ set_mempolicy(MPOL_INTERLEAVE, NULL, 0);
+#endif
}
return result;
* Re: Linux memory zone reclaim
2012-07-18 01:52 Linux memory zone reclaim Greg Smith <greg@2ndQuadrant.com>
2012-07-24 18:36 ` Re: Linux memory zone reclaim John Lister <john.lister@kickstone.com>
2012-07-24 18:41 ` Re: Linux memory zone reclaim Claudio Freire <klaussfreire@gmail.com>
2012-07-24 20:12 ` Re: Linux memory zone reclaim Claudio Freire <klaussfreire@gmail.com>
@ 2012-07-24 21:00 ` Claudio Freire <klaussfreire@gmail.com>
1 sibling, 0 replies; 19+ messages in thread
From: Claudio Freire @ 2012-07-24 21:00 UTC (permalink / raw)
To: John Lister <john.lister@kickstone.com>; +Cc: pgsql-performance
On Tue, Jul 24, 2012 at 5:12 PM, Claudio Freire <klaussfreire@gmail.com> wrote:
> Something like the attached patch (untested)
Sorry, on that patch, MPOL_INTERLEAVE should be MPOL_DEFAULT
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: Linux memory zone reclaim
2012-07-18 01:52 Linux memory zone reclaim Greg Smith <greg@2ndQuadrant.com>
2012-07-24 18:36 ` Re: Linux memory zone reclaim John Lister <john.lister@kickstone.com>
2012-07-24 18:41 ` Re: Linux memory zone reclaim Claudio Freire <klaussfreire@gmail.com>
2012-07-24 20:12 ` Re: Linux memory zone reclaim Claudio Freire <klaussfreire@gmail.com>
@ 2012-07-24 21:23 ` John Lister <john.lister@kickstone.com>
2012-07-24 21:37 ` Re: Linux memory zone reclaim Claudio Freire <klaussfreire@gmail.com>
2012-07-24 21:57 ` Re: Linux memory zone reclaim Scott Marlowe <scott.marlowe@gmail.com>
2012-07-24 22:05 ` Re: Linux memory zone reclaim Claudio Freire <klaussfreire@gmail.com>
1 sibling, 3 replies; 19+ messages in thread
From: John Lister @ 2012-07-24 21:23 UTC (permalink / raw)
To: Claudio Freire <klaussfreire@gmail.com>; +Cc: pgsql-performance
On 24/07/2012 21:12, Claudio Freire wrote:
> On Tue, Jul 24, 2012 at 3:41 PM, Claudio Freire <klaussfreire@gmail.com> wrote:
>> On Tue, Jul 24, 2012 at 3:36 PM, John Lister <john.lister@kickstone.com> wrote:
>>> Do you have a suggestion about how to do that? I'm running Ubuntu 12.04 and
>>> PG 9.1, I've modified pg_ctlcluster to cause pg_ctl to use a wrapper script
>>> which starts the postmaster using a numactl wrapper, but all subsequent
>>> client processes are started with interleaving enabled as well. Any ideas
>>> how to make just the postmaster process start with interleaving?
>> postmaster should call numactl right after forking:
>> http://linux.die.net/man/2/set_mempolicy
> Something like the attached patch (untested)
Cheers, I'll give it a go, I wonder if this is likely to be integrated
into the main code? As has been mentioned here before, postgresql isn't
as badly affected as mysql for example, but I'm wondering if the trend
to larger memory and more cores/nodes means it should be offered as an
option? Although saying that I've read that 10Gb of shared buffers may
be enough even in big machines 128+Gb ram..
Thoughts?
John
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: Linux memory zone reclaim
2012-07-18 01:52 Linux memory zone reclaim Greg Smith <greg@2ndQuadrant.com>
2012-07-24 18:36 ` Re: Linux memory zone reclaim John Lister <john.lister@kickstone.com>
2012-07-24 18:41 ` Re: Linux memory zone reclaim Claudio Freire <klaussfreire@gmail.com>
2012-07-24 20:12 ` Re: Linux memory zone reclaim Claudio Freire <klaussfreire@gmail.com>
2012-07-24 21:23 ` Re: Linux memory zone reclaim John Lister <john.lister@kickstone.com>
@ 2012-07-24 21:37 ` Claudio Freire <klaussfreire@gmail.com>
2 siblings, 0 replies; 19+ messages in thread
From: Claudio Freire @ 2012-07-24 21:37 UTC (permalink / raw)
To: John Lister <john.lister@kickstone.com>; +Cc: pgsql-performance
On Tue, Jul 24, 2012 at 6:23 PM, John Lister <john.lister@kickstone.com> wrote:
> Cheers, I'll give it a go, I wonder if this is likely to be integrated into
> the main code? As has been mentioned here before, postgresql isn't as badly
> affected as mysql for example, but I'm wondering if the trend to larger
> memory and more cores/nodes means it should be offered as an option?
> Although saying that I've read that 10Gb of shared buffers may be enough
> even in big machines 128+Gb ram..
Remember to change MPOL_INTERLEAVED to MPOL_DEFAULT ;-)
I'm trying to test it myself
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: Linux memory zone reclaim
2012-07-18 01:52 Linux memory zone reclaim Greg Smith <greg@2ndQuadrant.com>
2012-07-24 18:36 ` Re: Linux memory zone reclaim John Lister <john.lister@kickstone.com>
2012-07-24 18:41 ` Re: Linux memory zone reclaim Claudio Freire <klaussfreire@gmail.com>
2012-07-24 20:12 ` Re: Linux memory zone reclaim Claudio Freire <klaussfreire@gmail.com>
2012-07-24 21:23 ` Re: Linux memory zone reclaim John Lister <john.lister@kickstone.com>
@ 2012-07-24 21:57 ` Scott Marlowe <scott.marlowe@gmail.com>
2 siblings, 0 replies; 19+ messages in thread
From: Scott Marlowe @ 2012-07-24 21:57 UTC (permalink / raw)
To: John Lister <john.lister@kickstone.com>; +Cc: Claudio Freire <klaussfreire@gmail.com>; pgsql-performance
My experience is that disabling swap and turning off zone_reclaim_mode
gets rid of any real problem for a large memory postgresql database
server. While it would be great to have a NUMA aware pgsql, I
question the solidity and reliability of the current linux kernel
implementation in a NUMA evironment, especially given the poor
behaviour of the linux kernel as regards swap behaviour.
On Tue, Jul 24, 2012 at 3:23 PM, John Lister <john.lister@kickstone.com> wrote:
> On 24/07/2012 21:12, Claudio Freire wrote:
>>
>> On Tue, Jul 24, 2012 at 3:41 PM, Claudio Freire <klaussfreire@gmail.com>
>> wrote:
>>>
>>> On Tue, Jul 24, 2012 at 3:36 PM, John Lister <john.lister@kickstone.com>
>>> wrote:
>>>>
>>>> Do you have a suggestion about how to do that? I'm running Ubuntu 12.04
>>>> and
>>>> PG 9.1, I've modified pg_ctlcluster to cause pg_ctl to use a wrapper
>>>> script
>>>> which starts the postmaster using a numactl wrapper, but all subsequent
>>>> client processes are started with interleaving enabled as well. Any
>>>> ideas
>>>> how to make just the postmaster process start with interleaving?
>>>
>>> postmaster should call numactl right after forking:
>>> http://linux.die.net/man/2/set_mempolicy
>>
>> Something like the attached patch (untested)
>
> Cheers, I'll give it a go, I wonder if this is likely to be integrated into
> the main code? As has been mentioned here before, postgresql isn't as badly
> affected as mysql for example, but I'm wondering if the trend to larger
> memory and more cores/nodes means it should be offered as an option?
> Although saying that I've read that 10Gb of shared buffers may be enough
> even in big machines 128+Gb ram..
>
> Thoughts?
>
> John
>
>
>
>
> --
> Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-performance
--
To understand recursion, one must first understand recursion.
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: Linux memory zone reclaim
2012-07-18 01:52 Linux memory zone reclaim Greg Smith <greg@2ndQuadrant.com>
2012-07-24 18:36 ` Re: Linux memory zone reclaim John Lister <john.lister@kickstone.com>
2012-07-24 18:41 ` Re: Linux memory zone reclaim Claudio Freire <klaussfreire@gmail.com>
2012-07-24 20:12 ` Re: Linux memory zone reclaim Claudio Freire <klaussfreire@gmail.com>
2012-07-24 21:23 ` Re: Linux memory zone reclaim John Lister <john.lister@kickstone.com>
@ 2012-07-24 22:05 ` Claudio Freire <klaussfreire@gmail.com>
2 siblings, 0 replies; 19+ messages in thread
From: Claudio Freire @ 2012-07-24 22:05 UTC (permalink / raw)
To: John Lister <john.lister@kickstone.com>; +Cc: pgsql-performance
On Tue, Jul 24, 2012 at 6:23 PM, John Lister <john.lister@kickstone.com> wrote:
> Cheers, I'll give it a go, I wonder if this is likely to be integrated into
> the main code? As has been mentioned here before, postgresql isn't as badly
> affected as mysql for example, but I'm wondering if the trend to larger
> memory and more cores/nodes means it should be offered as an option?
> Although saying that I've read that 10Gb of shared buffers may be enough
> even in big machines 128+Gb ram..
>
> Thoughts?
The attached (better) patch builds and doesn't crash at least.
Which is always good.
Configure with --with-numa
Attachments:
numa.patch (application/octet-stream, 3.6K) download
diff --git a/configure.in b/configure.in
index 8ed09f5..44eff38 100644
--- a/configure.in
+++ b/configure.in
@@ -19,10 +19,10 @@ m4_pattern_forbid(^PGAC_)dnl to catch undefined macros
AC_INIT([PostgreSQL], [9.3devel], [pgsql-bugs@postgresql.org])
-m4_if(m4_defn([m4_PACKAGE_VERSION]), [2.63], [], [m4_fatal([Autoconf version 2.63 is required.
-Untested combinations of 'autoconf' and PostgreSQL versions are not
-recommended. You can remove the check from 'configure.in' but it is then
-your responsibility whether the result works or not.])])
+#m4_if(m4_defn([m4_PACKAGE_VERSION]), [2.63], [], [m4_fatal([Autoconf version 2.63 is required.
+#Untested combinations of 'autoconf' and PostgreSQL versions are not
+#recommended. You can remove the check from 'configure.in' but it is then
+#your responsibility whether the result works or not.])])
AC_COPYRIGHT([Copyright (c) 1996-2012, PostgreSQL Global Development Group])
AC_CONFIG_SRCDIR([src/backend/access/common/heaptuple.c])
AC_CONFIG_AUX_DIR(config)
@@ -604,6 +604,17 @@ PGAC_ARG_BOOL(with, gssapi, no, [build with GSSAPI support],
AC_MSG_RESULT([$with_gssapi])
#
+# NUMA
+#
+AC_MSG_CHECKING([whether to build with NUMA support])
+PGAC_ARG_BOOL(with, numa, no, [build with NUMA support],
+[
+ AC_DEFINE(ENABLE_NUMA, 1, [Define to build with NUMA support. (--with-numa)])
+])
+AC_MSG_RESULT([$with_numa])
+AC_SUBST(with_numa)
+
+#
# Kerberos 5
#
AC_MSG_CHECKING([whether to build with Kerberos 5 support])
@@ -922,6 +933,15 @@ if test "$with_gssapi" = yes ; then
fi
fi
+if test "$with_numa" = yes ; then
+ if test "$PORTNAME" != "win32"; then
+ AC_SEARCH_LIBS(set_mempolicy, [numa], [],
+ [AC_MSG_ERROR([could not find function 'set_mempolicy' required for NUMA])])
+ else
+ LIBS="$LIBS -lnuma"
+ fi
+fi
+
if test "$with_krb5" = yes ; then
if test "$PORTNAME" != "win32"; then
AC_SEARCH_LIBS(com_err, [krb5 'krb5 -lcrypto -ldes -lasn1 -lroken' com_err 'com_err -lssl -lcrypto'], [],
diff --git a/src/backend/postmaster/fork_process.c b/src/backend/postmaster/fork_process.c
index 2a415e8..da5ab0c 100644
--- a/src/backend/postmaster/fork_process.c
+++ b/src/backend/postmaster/fork_process.c
@@ -18,6 +18,10 @@
#include <sys/time.h>
#include <unistd.h>
+#ifdef ENABLE_NUMA
+#include <numaif.h>
+#endif
+
#ifndef WIN32
/*
* Wrapper for fork(). Return values are the same as those for fork():
@@ -124,6 +128,10 @@ fork_process(void)
}
}
#endif /* LINUX_OOM_ADJ */
+
+#ifdef ENABLE_NUMA
+ set_mempolicy(MPOL_DEFAULT, NULL, 0);
+#endif
}
return result;
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index bbf725d..f2a2e1e 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -91,6 +91,10 @@
#include <dns_sd.h>
#endif
+#ifdef ENABLE_NUMA
+#include <numaif.h>
+#endif
+
#include "access/transam.h"
#include "access/xlog.h"
#include "bootstrap/bootstrap.h"
@@ -492,6 +496,19 @@ int postmaster_alive_fds[2] = {-1, -1};
HANDLE PostmasterHandle;
#endif
+#ifdef ENABLE_NUMA
+static void
+NumaInterleave(void)
+{
+ int mode;
+ unsigned long nodemask[64];
+
+ if (0 == get_mempolicy(&mode, nodemask, sizeof(nodemask)*8, 0, 0)) {
+ set_mempolicy(MPOL_INTERLEAVE, nodemask, sizeof(nodemask)*8);
+ }
+}
+#endif
+
/*
* Postmaster main entry point
*/
@@ -515,6 +532,13 @@ PostmasterMain(int argc, char *argv[])
*/
umask(S_IRWXG | S_IRWXO);
+#ifdef ENABLE_NUMA
+ /*
+ * interleave shared memory
+ */
+ NumaInterleave();
+#endif
+
/*
* Fire up essential subsystems: memory management
*/
* Re: Linux memory zone reclaim
2012-07-18 01:52 Linux memory zone reclaim Greg Smith <greg@2ndQuadrant.com>
@ 2012-07-27 00:25 ` Bruce Momjian <bruce@momjian.us>
3 siblings, 0 replies; 19+ messages in thread
From: Bruce Momjian @ 2012-07-27 00:25 UTC (permalink / raw)
To: Greg Smith <greg@2ndQuadrant.com>; +Cc: pgsql-performance
On Tue, Jul 17, 2012 at 09:52:11PM -0400, Greg Smith wrote:
> I've taken to disabling /proc/sys/vm/zone_reclaim_mode on any Linux
> system where it's turned on now. I'm still working through whether
--------------------------------
> it also makes sense in all cases to use the more complicated memory
> interleaving suggestions that MySQL users have implemented,
> something most people would need to push into their PostgreSQL
> server started up scripts in /etc/init.d (That will be a fun
> rpm/deb packaging issue to deal with if this becomes more
> wide-spread) Suggestions on whether that is necessary, or if just
> disabling zone_reclaim is enough, are welcome from anyone who wants
> to try and benchmark it.
Should I be turning it off on my server too? It is enabled on my
system.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ It's impossible for everything to be true. +
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: Linux memory zone reclaim
2012-07-18 01:52 Linux memory zone reclaim Greg Smith <greg@2ndQuadrant.com>
@ 2012-07-30 16:43 ` Kevin Grittner <Kevin.Grittner@wicourts.gov>
2012-07-30 17:09 ` Re: Linux memory zone reclaim Scott Marlowe <scott.marlowe@gmail.com>
2012-07-30 17:26 ` Re: Linux memory zone reclaim Bruce Momjian <bruce@momjian.us>
3 siblings, 2 replies; 19+ messages in thread
From: Kevin Grittner @ 2012-07-30 16:43 UTC (permalink / raw)
To: Greg Smith <greg@2ndQuadrant.com>; pgsql-performance
Greg Smith <greg@2ndQuadrant.com> wrote:
> You can tell if this is turned on like this:
>
> echo /proc/sys/vm/zone_reclaim_mode
As a data point, the benchmarks I did for some of the 9.2
scalability features does not appear to have this turned on:
# cat /proc/sys/vm/zone_reclaim_mode
0
Our Linux version:
Linux version 2.6.32.46-0.3-default (geeko@buildhost) (gcc version
4.3.4 [gcc-4_3-branch revision 152973] (SUSE Linux) ) #1 SMP
2011-09-29 17:49:31 +0200
This has 32 cores (64 "threads" with HT) on 4 Xeon X7560 CPUs.
Intel(R) Xeon(R) CPU X7560 @ 2.27GHz
It has 256GB RAM on 4GB DIMMs, with each core controlling 2 DIMMs
and each core able to directly talk to every other core. So, it
is non-uniform, but with this arrangement it is more a matter that
there is an 8GB set of memory that is "fast" for each core and the
other 97% of RAM is all accessible at the same speed. There were
some other options for what to install on this system or how to
install it which wouldn't have kept things this tight.
> Install the numactl utility and you can see why it's made that
> decision:
We get this:
# numactl --hardware
available: 4 nodes (0-3)
node 0 cpus: 0 1 2 3 4 5 6 7 32 33 34 35 36 37 38 39
node 0 size: 65519 MB
node 0 free: 283 MB
node 1 cpus: 8 9 10 11 12 13 14 15 40 41 42 43 44 45 46 47
node 1 size: 65536 MB
node 1 free: 25 MB
node 2 cpus: 16 17 18 19 20 21 22 23 48 49 50 51 52 53 54 55
node 2 size: 65536 MB
node 2 free: 26 MB
node 3 cpus: 24 25 26 27 28 29 30 31 56 57 58 59 60 61 62 63
node 3 size: 65536 MB
node 3 free: 25 MB
node distances:
node 0 1 2 3
0: 10 11 11 11
1: 11 10 11 11
2: 11 11 10 11
3: 11 11 11 10
When considering a hardware purchase, it might be wise to pay close
attention to how "far" a core may need to go to get to the most
"distant" RAM.
-Kevin
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: Linux memory zone reclaim
2012-07-18 01:52 Linux memory zone reclaim Greg Smith <greg@2ndQuadrant.com>
2012-07-30 16:43 ` Re: Linux memory zone reclaim Kevin Grittner <Kevin.Grittner@wicourts.gov>
@ 2012-07-30 17:09 ` Scott Marlowe <scott.marlowe@gmail.com>
2012-08-03 22:30 ` Re: Linux memory zone reclaim Josh Berkus <josh@agliodbs.com>
1 sibling, 1 reply; 19+ messages in thread
From: Scott Marlowe @ 2012-07-30 17:09 UTC (permalink / raw)
To: Kevin Grittner <Kevin.Grittner@wicourts.gov>; +Cc: Greg Smith <greg@2ndquadrant.com>; pgsql-performance
On Mon, Jul 30, 2012 at 10:43 AM, Kevin Grittner
<Kevin.Grittner@wicourts.gov> wrote:
> node distances:
> node 0 1 2 3
> 0: 10 11 11 11
> 1: 11 10 11 11
> 2: 11 11 10 11
> 3: 11 11 11 10
>
> When considering a hardware purchase, it might be wise to pay close
> attention to how "far" a core may need to go to get to the most
> "distant" RAM.
I think the zone_reclaim gets turned on with a high ratio. If the
inter node costs were the same, and the intranode costs dropped in
half, zone reclaim would likely get turned on at boot time.
I had something similar in a 48 core system but if I recall correctly
the matrix was 8x8 and the cost differential was much higher.
The symptoms I saw was that a very hard working db, on a 128G machine
with about 95G as OS / kernel cache, would slow to a crawl with kswapd
working very hard (I think it was kswapd) after a period of 1 to 3
weeks. Note that actual swap in and out wasn't all that great by
vmstat. The same performance hit happened on a similar machine used
as a file server after a similar period of warm up.
The real danger here is that the misbehavior can take a long time to
show up, and from what I read at the time, the performance gain for
any zone reclaim = 1 was minimal for a file or db server, and more in
line for a large virtual machine farm, with a lot of processes chopped
into sections small enough to fit in one node's memory and not need a
lot of access from another node. Anything that relies on the OS to
cache is likely not served by zone reclaim = 1.
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: Linux memory zone reclaim
2012-07-18 01:52 Linux memory zone reclaim Greg Smith <greg@2ndQuadrant.com>
2012-07-30 16:43 ` Re: Linux memory zone reclaim Kevin Grittner <Kevin.Grittner@wicourts.gov>
2012-07-30 17:09 ` Re: Linux memory zone reclaim Scott Marlowe <scott.marlowe@gmail.com>
@ 2012-08-03 22:30 ` Josh Berkus <josh@agliodbs.com>
2012-08-03 23:02 ` Re: Linux memory zone reclaim Scott Marlowe <scott.marlowe@gmail.com>
0 siblings, 1 reply; 19+ messages in thread
From: Josh Berkus @ 2012-08-03 22:30 UTC (permalink / raw)
To: pgsql-performance
On 7/30/12 10:09 AM, Scott Marlowe wrote:
> I think the zone_reclaim gets turned on with a high ratio. If the
> inter node costs were the same, and the intranode costs dropped in
> half, zone reclaim would likely get turned on at boot time.
We've been seeing a major problem with zone_reclaim and Linux, in that
Linux won't use the FS cache on the "distant" RAM *at all* if it thinks
that RAM is distant enough. Thus, you get instances of seeing only half
of RAM used for FS cache, even though the database is 5X larger than RAM.
This is poor design on Linux's part, since even the distant RAM is
faster than disk. For now, we've been disabling zone_reclaim entirely.
--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: Linux memory zone reclaim
2012-07-18 01:52 Linux memory zone reclaim Greg Smith <greg@2ndQuadrant.com>
2012-07-30 16:43 ` Re: Linux memory zone reclaim Kevin Grittner <Kevin.Grittner@wicourts.gov>
2012-07-30 17:09 ` Re: Linux memory zone reclaim Scott Marlowe <scott.marlowe@gmail.com>
2012-08-03 22:30 ` Re: Linux memory zone reclaim Josh Berkus <josh@agliodbs.com>
@ 2012-08-03 23:02 ` Scott Marlowe <scott.marlowe@gmail.com>
2012-08-03 23:15 ` Re: Linux memory zone reclaim Josh Berkus <josh@agliodbs.com>
0 siblings, 1 reply; 19+ messages in thread
From: Scott Marlowe @ 2012-08-03 23:02 UTC (permalink / raw)
To: Josh Berkus <josh@agliodbs.com>; +Cc: pgsql-performance
On Fri, Aug 3, 2012 at 4:30 PM, Josh Berkus <josh@agliodbs.com> wrote:
> On 7/30/12 10:09 AM, Scott Marlowe wrote:
>> I think the zone_reclaim gets turned on with a high ratio. If the
>> inter node costs were the same, and the intranode costs dropped in
>> half, zone reclaim would likely get turned on at boot time.
>
> We've been seeing a major problem with zone_reclaim and Linux, in that
> Linux won't use the FS cache on the "distant" RAM *at all* if it thinks
> that RAM is distant enough. Thus, you get instances of seeing only half
> of RAM used for FS cache, even though the database is 5X larger than RAM.
>
> This is poor design on Linux's part, since even the distant RAM is
> faster than disk. For now, we've been disabling zone_reclaim entirely.
I haven't run into this, but we were running ubuntu 10.04 LTS. What
kernel were you running when this happened? I'd love to see a test
case on this, as it seems like a major regression if it's on newer
kernels, and we're looking at running 12.04 LTS soon on one of our
bigger machines.
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: Linux memory zone reclaim
2012-07-18 01:52 Linux memory zone reclaim Greg Smith <greg@2ndQuadrant.com>
2012-07-30 16:43 ` Re: Linux memory zone reclaim Kevin Grittner <Kevin.Grittner@wicourts.gov>
2012-07-30 17:09 ` Re: Linux memory zone reclaim Scott Marlowe <scott.marlowe@gmail.com>
2012-08-03 22:30 ` Re: Linux memory zone reclaim Josh Berkus <josh@agliodbs.com>
2012-08-03 23:02 ` Re: Linux memory zone reclaim Scott Marlowe <scott.marlowe@gmail.com>
@ 2012-08-03 23:15 ` Josh Berkus <josh@agliodbs.com>
0 siblings, 0 replies; 19+ messages in thread
From: Josh Berkus @ 2012-08-03 23:15 UTC (permalink / raw)
To: Scott Marlowe <scott.marlowe@gmail.com>; +Cc: pgsql-performance
>> This is poor design on Linux's part, since even the distant RAM is
>> faster than disk. For now, we've been disabling zone_reclaim entirely.
>
> I haven't run into this, but we were running ubuntu 10.04 LTS. What
> kernel were you running when this happened? I'd love to see a test
> case on this, as it seems like a major regression if it's on newer
> kernels, and we're looking at running 12.04 LTS soon on one of our
> bigger machines.
Jeff Frost will have a blog up about it later; we're still collecting data.
--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: Linux memory zone reclaim
2012-07-18 01:52 Linux memory zone reclaim Greg Smith <greg@2ndQuadrant.com>
2012-07-30 16:43 ` Re: Linux memory zone reclaim Kevin Grittner <Kevin.Grittner@wicourts.gov>
@ 2012-07-30 17:26 ` Bruce Momjian <bruce@momjian.us>
1 sibling, 0 replies; 19+ messages in thread
From: Bruce Momjian @ 2012-07-30 17:26 UTC (permalink / raw)
To: Kevin Grittner <Kevin.Grittner@wicourts.gov>; +Cc: Greg Smith <greg@2ndQuadrant.com>; pgsql-performance
> node distances:
> node 0 1 2 3
> 0: 10 11 11 11
> 1: 11 10 11 11
> 2: 11 11 10 11
> 3: 11 11 11 10
>
> When considering a hardware purchase, it might be wise to pay close
> attention to how "far" a core may need to go to get to the most
> "distant" RAM.
Yikes, my server is certainly asymmetric:
node distances:
node 0 1
0: 10 21
1: 21 10
and my Debian Squeeze certainly knows that:
$ cat < /proc/sys/vm/zone_reclaim_mode
1
Server specs:
http://momjian.us/main/blogs/pgblog/2012.html#January_20_2012
I have 12 2GB DDR3 DIMs.
Of course, my home server is ridiculously idle too. :-)
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ It's impossible for everything to be true. +
^ permalink raw reply [nested|flat] 19+ messages in thread
end of thread, other threads:[~2012-08-03 23:15 UTC | newest]
Thread overview: 19+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2012-07-18 01:52 Linux memory zone reclaim Greg Smith <greg@2ndQuadrant.com>
2012-07-18 02:00 ` Scott Marlowe <scott.marlowe@gmail.com>
2012-07-18 03:51 ` Dave Crooke <dcrooke@gmail.com>
2012-07-18 05:38 ` Claudio Freire <klaussfreire@gmail.com>
2012-07-24 18:36 ` John Lister <john.lister@kickstone.com>
2012-07-24 18:41 ` Claudio Freire <klaussfreire@gmail.com>
2012-07-24 20:12 ` Claudio Freire <klaussfreire@gmail.com>
2012-07-24 21:00 ` Claudio Freire <klaussfreire@gmail.com>
2012-07-24 21:23 ` John Lister <john.lister@kickstone.com>
2012-07-24 21:37 ` Claudio Freire <klaussfreire@gmail.com>
2012-07-24 21:57 ` Scott Marlowe <scott.marlowe@gmail.com>
2012-07-24 22:05 ` Claudio Freire <klaussfreire@gmail.com>
2012-07-27 00:25 ` Bruce Momjian <bruce@momjian.us>
2012-07-30 16:43 ` Kevin Grittner <Kevin.Grittner@wicourts.gov>
2012-07-30 17:09 ` Scott Marlowe <scott.marlowe@gmail.com>
2012-08-03 22:30 ` Josh Berkus <josh@agliodbs.com>
2012-08-03 23:02 ` Scott Marlowe <scott.marlowe@gmail.com>
2012-08-03 23:15 ` Josh Berkus <josh@agliodbs.com>
2012-07-30 17:26 ` Bruce Momjian <bruce@momjian.us>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox