Message-ID: <B994348460014EC7BA65F310E9D30EBD@BERNICE>
From: "Midge Brown" <midgems@sbcglobal.net>
To: <sthomas@optionshouse.com>,
	<pgsql-performance@postgresql.org>
References: <50E4AAB1.9040902@optionshouse.com>
Subject: Re: Two Necessary Kernel Tweaks for Linux Systems
Date: Tue, 8 Jan 2013 10:25:52 -0800
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_NextPart_000_005E_01CDED8A.89169240"
Precedence: bulk
Sender: pgsql-performance-owner@postgresql.org

This is a multi-part message in MIME format.

------=_NextPart_000_005E_01CDED8A.89169240
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

The kernel on our Linux system doesn't appear to have these two settings =
according to the list provided by sysctl -a. Please pardon my ignorance, =
but should I add them?=20

We have Postgresql 9.0 on Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 =
EDT 2009 x86_64 x86_64 x86_64 GNU/Linux

Thanks,
Midge

  ----- Original Message -----=20
  From: Shaun Thomas=20
  To: pgsql-performance@postgresql.org=20
  Sent: Wednesday, January 02, 2013 1:46 PM
  Subject: [PERFORM] Two Necessary Kernel Tweaks for Linux Systems


  Hey everyone!

  After much testing and hair-pulling, we've confirmed two kernel =
settings=20
  that should always be modified in production Linux systems. Especially =

  new ones with the completely fair scheduler (CFS) as opposed to the =
O(1)=20
  scheduler.

  If you want to follow along, these are:

  /proc/sys/kernel/sched_migration_cost
  /proc/sys/kernel/sched_autogroup_enabled

  Which correspond to sysctl settings:

  kernel.sched_migration_cost
  kernel.sched_autogroup_enabled

  What do these settings do?
  --------------------------

  * sched_migration_cost

  The migration cost is the total time the scheduler will consider a=20
  migrated process "cache hot" and thus less likely to be re-migrated. =
By=20
  default, this is 0.5ms (500000 ns), and as the size of the process =
table=20
  increases, eventually causes the scheduler to break down. On our=20
  systems, after a smooth degradation with increasing connection count,=20
  system CPU spiked from 20 to 70% sustained and TPS was cut by 5-10x =
once=20
  we crossed some invisible connection count threshold. For us, that was =
a=20
  pgbench with 900 or more clients.

  The migration cost should be increased, almost universally on server=20
  systems with many processes. This means systems like PostgreSQL or=20
  Apache would benefit from having higher migration costs. We've had =
good=20
  luck with a setting of 5ms (5000000 ns) instead.

  When the breakdown occurs, system CPU (as obtained from sar) increases =

  from 20% on a heavy pgbench (scale 3500 on a 72GB system) to over 70%, =

  and %nice/%user is cut by half or more. A higher migration cost=20
  essentially eliminates this artificial throttle.

  * sched_autogroup_enabled

  This is a relatively new patch which Linus lauded back in late 2010. =
It=20
  basically groups tasks by TTY so perceived responsiveness is improved. =

  But on server systems, large daemons like PostgreSQL are going to be=20
  launched from the same pseudo-TTY, and be effectively choked out of =
CPU=20
  cycles in favor of less important tasks.

  The default setting is 1 (enabled) on some platforms. By setting this =
to=20
  0 (disabled), we saw an outright 30% performance boost on the same=20
  pgbench test. A fully cached scale 3500 database on a 72GB system went =

  from 67k TPS to 82k TPS with 900 client connections.

  Total Benefit
  -------------

  At higher connections counts, such as systems that can't use pooling =
or=20
  make extensive use of prepared queries, these can massively affect=20
  performance. At 900 connections, our test systems were at 17k TPS=20
  unaltered, but 85k TPS after these two modifications. Even with this=20
  performance boost, we still had 40% CPU free instead of 0%. In effect, =

  the logarithmic performance of the new scheduler is returned to normal =

  under large process tables.

  Some systems will have a higher "cracking" point than others. The =
effect=20
  is amplified when a system is under high memory pressure, hence a lot =
of=20
  expensive queries on a high number of concurrent connections is the=20
  easiest way to replicate these results.

  Admins migrating from older systems (RHEL 5.x) may find this =
especially=20
  shocking, because the old O(1) scheduler was too "stupid" to have =
these=20
  advanced features, hence it was impossible to cause this kind of =
behavior.

  There's probably still a little room for improvement here, since =
30-40%=20
  CPU is still unclaimed in our larger tests. I'd like to see the total=20
  performance drop (175k ideal TPS at 24-connections) decreased. But =
these=20
  kernel tweaks are rarely discussed anywhere, it seems. There doesn't=20
  seem to be any consensus on how these (and other) scheduler settings=20
  should be modified under different usage scenarios.

  I just figured I'd share, since we found this info so beneficial.

  --=20
  Shaun Thomas
  OptionsHouse | 141 W. Jackson Blvd. | Suite 500 | Chicago IL, 60604
  312-444-8534
  sthomas@optionshouse.com

  ______________________________________________

  See http://www.peak6.com/email_disclaimer/ for terms and conditions =
related to this email


  --=20
  Sent via pgsql-performance mailing list =
(pgsql-performance@postgresql.org)
  To make changes to your subscription:
  http://www.postgresql.org/mailpref/pgsql-performance
------=_NextPart_000_005E_01CDED8A.89169240
Content-Type: text/html;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META content=3D"text/html; charset=3Diso-8859-1" =
http-equiv=3DContent-Type>
<META name=3DGENERATOR content=3D"MSHTML 8.00.6001.19393">
<STYLE></STYLE>
</HEAD>
<BODY bgColor=3D#ffffff>
<DIV><FONT size=3D2 face=3DCourier>The kernel on our Linux system =
doesn't appear to=20
have these two settings according to the list provided by sysctl -a. =
Please=20
pardon my ignorance, but should I add them? </FONT></DIV>
<DIV><FONT size=3D2 face=3DCourier></FONT>&nbsp;</DIV>
<DIV><FONT size=3D2 face=3DCourier>We have Postgresql 9.0 on Linux =
2.6.18-164.el5 #1=20
SMP Thu Sep 3 03:28:30 EDT 2009 x86_64 x86_64 x86_64 =
GNU/Linux</FONT></DIV>
<DIV><FONT size=3D2 face=3DCourier></FONT>&nbsp;</DIV>
<DIV><FONT size=3D2 face=3DCourier>Thanks,</FONT></DIV>
<DIV><FONT size=3D2 face=3DCourier>Midge</FONT></DIV>
<DIV><FONT size=3D2 face=3DCourier></FONT>&nbsp;</DIV>
<BLOCKQUOTE=20
style=3D"BORDER-LEFT: #000000 2px solid; PADDING-LEFT: 5px; =
PADDING-RIGHT: 0px; MARGIN-LEFT: 5px; MARGIN-RIGHT: 0px">
  <DIV style=3D"FONT: 10pt arial">----- Original Message ----- </DIV>
  <DIV=20
  style=3D"FONT: 10pt arial; BACKGROUND: #e4e4e4; font-color: =
black"><B>From:</B>=20
  <A title=3Dsthomas@optionshouse.com =
href=3D"mailto:sthomas@optionshouse.com">Shaun=20
  Thomas</A> </DIV>
  <DIV style=3D"FONT: 10pt arial"><B>To:</B> <A=20
  title=3Dpgsql-performance@postgresql.org=20
  =
href=3D"mailto:pgsql-performance@postgresql.org">pgsql-performance@postgr=
esql.org</A>=20
  </DIV>
  <DIV style=3D"FONT: 10pt arial"><B>Sent:</B> Wednesday, January 02, =
2013 1:46=20
  PM</DIV>
  <DIV style=3D"FONT: 10pt arial"><B>Subject:</B> [PERFORM] Two =
Necessary Kernel=20
  Tweaks for Linux Systems</DIV>
  <DIV><BR></DIV>Hey everyone!<BR><BR>After much testing and =
hair-pulling, we've=20
  confirmed two kernel settings <BR>that should always be modified in =
production=20
  Linux systems. Especially <BR>new ones with the completely fair =
scheduler=20
  (CFS) as opposed to the O(1) <BR>scheduler.<BR><BR>If you want to =
follow=20
  along, these=20
  =
are:<BR><BR>/proc/sys/kernel/sched_migration_cost<BR>/proc/sys/kernel/sch=
ed_autogroup_enabled<BR><BR>Which=20
  correspond to sysctl=20
  =
settings:<BR><BR>kernel.sched_migration_cost<BR>kernel.sched_autogroup_en=
abled<BR><BR>What=20
  do these settings do?<BR>--------------------------<BR><BR>*=20
  sched_migration_cost<BR><BR>The migration cost is the total time the =
scheduler=20
  will consider a <BR>migrated process "cache hot" and thus less likely =
to be=20
  re-migrated. By <BR>default, this is 0.5ms (500000 ns), and as the =
size of the=20
  process table <BR>increases, eventually causes the scheduler to break =
down. On=20
  our <BR>systems, after a smooth degradation with increasing connection =
count,=20
  <BR>system CPU spiked from 20 to 70% sustained and TPS was cut by =
5-10x once=20
  <BR>we crossed some invisible connection count threshold. For us, that =
was a=20
  <BR>pgbench with 900 or more clients.<BR><BR>The migration cost should =
be=20
  increased, almost universally on server <BR>systems with many =
processes. This=20
  means systems like PostgreSQL or <BR>Apache would benefit from having =
higher=20
  migration costs. We've had good <BR>luck with a setting of 5ms =
(5000000 ns)=20
  instead.<BR><BR>When the breakdown occurs, system CPU (as obtained =
from sar)=20
  increases <BR>from 20% on a heavy pgbench (scale 3500 on a 72GB =
system) to=20
  over 70%, <BR>and %nice/%user is cut by half or more. A higher =
migration cost=20
  <BR>essentially eliminates this artificial throttle.<BR><BR>*=20
  sched_autogroup_enabled<BR><BR>This is a relatively new patch which =
Linus=20
  lauded back in late 2010. It <BR>basically groups tasks by TTY so =
perceived=20
  responsiveness is improved. <BR>But on server systems, large daemons =
like=20
  PostgreSQL are going to be <BR>launched from the same pseudo-TTY, and =
be=20
  effectively choked out of CPU <BR>cycles in favor of less important=20
  tasks.<BR><BR>The default setting is 1 (enabled) on some platforms. By =
setting=20
  this to <BR>0 (disabled), we saw an outright 30% performance boost on =
the same=20
  <BR>pgbench test. A fully cached scale 3500 database on a 72GB system =
went=20
  <BR>from 67k TPS to 82k TPS with 900 client connections.<BR><BR>Total=20
  Benefit<BR>-------------<BR><BR>At higher connections counts, such as =
systems=20
  that can't use pooling or <BR>make extensive use of prepared queries, =
these=20
  can massively affect <BR>performance. At 900 connections, our test =
systems=20
  were at 17k TPS <BR>unaltered, but 85k TPS after these two =
modifications. Even=20
  with this <BR>performance boost, we still had 40% CPU free instead of =
0%. In=20
  effect, <BR>the logarithmic performance of the new scheduler is =
returned to=20
  normal <BR>under large process tables.<BR><BR>Some systems will have a =
higher=20
  "cracking" point than others. The effect <BR>is amplified when a =
system is=20
  under high memory pressure, hence a lot of <BR>expensive queries on a =
high=20
  number of concurrent connections is the <BR>easiest way to replicate =
these=20
  results.<BR><BR>Admins migrating from older systems (RHEL 5.x) may =
find this=20
  especially <BR>shocking, because the old O(1) scheduler was too =
"stupid" to=20
  have these <BR>advanced features, hence it was impossible to cause =
this kind=20
  of behavior.<BR><BR>There's probably still a little room for =
improvement here,=20
  since 30-40% <BR>CPU is still unclaimed in our larger tests. I'd like =
to see=20
  the total <BR>performance drop (175k ideal TPS at 24-connections) =
decreased.=20
  But these <BR>kernel tweaks are rarely discussed anywhere, it seems. =
There=20
  doesn't <BR>seem to be any consensus on how these (and other) =
scheduler=20
  settings <BR>should be modified under different usage =
scenarios.<BR><BR>I just=20
  figured I'd share, since we found this info so beneficial.<BR><BR>-- =
<BR>Shaun=20
  Thomas<BR>OptionsHouse | 141 W. Jackson Blvd. | Suite 500 | Chicago =
IL,=20
  60604<BR>312-444-8534<BR><A=20
  =
href=3D"mailto:sthomas@optionshouse.com">sthomas@optionshouse.com</A><BR>=
<BR>______________________________________________<BR><BR>See=20
  <A=20
  =
href=3D"http://www.peak6.com/email_disclaimer/">http://www.peak6.com/emai=
l_disclaimer/</A>=20
  for terms and conditions related to this email<BR><BR><BR>-- <BR>Sent =
via=20
  pgsql-performance mailing list (<A=20
  =
href=3D"mailto:pgsql-performance@postgresql.org">pgsql-performance@postgr=
esql.org</A>)<BR>To=20
  make changes to your subscription:<BR><A=20
  =
href=3D"http://www.postgresql.org/mailpref/pgsql-performance">http://www.=
postgresql.org/mailpref/pgsql-performance</A></BLOCKQUOTE></BODY></HTML>

------=_NextPart_000_005E_01CDED8A.89169240--