pg.ddx.io pgsql-performance@postgresql.org mailing list archive
help / color / mirror / Atom feedTwo Necessary Kernel Tweaks for Linux Systems
17+ messages / 10 participants
[nested] [flat]
* Two Necessary Kernel Tweaks for Linux Systems
@ 2013-01-02 21:46 Shaun Thomas <sthomas@optionshouse.com>
2013-01-03 00:47 ` Re: Two Necessary Kernel Tweaks for Linux Systems Richard Neill <rn214@richardneill.org>
2013-01-07 19:22 ` Re: Two Necessary Kernel Tweaks for Linux Systems Merlin Moncure <mmoncure@gmail.com>
2013-01-08 08:29 ` Re: Two Necessary Kernel Tweaks for Linux Systems Andrea Suisani <sickpig@opinioni.net>
2013-01-08 18:25 ` Re: Two Necessary Kernel Tweaks for Linux Systems Midge Brown <midgems@sbcglobal.net>
0 siblings, 4 replies; 17+ messages in thread
From: Shaun Thomas @ 2013-01-02 21:46 UTC (permalink / raw)
To: pgsql-performance
Hey everyone!
After much testing and hair-pulling, we've confirmed two kernel settings
that should always be modified in production Linux systems. Especially
new ones with the completely fair scheduler (CFS) as opposed to the O(1)
scheduler.
If you want to follow along, these are:
/proc/sys/kernel/sched_migration_cost
/proc/sys/kernel/sched_autogroup_enabled
Which correspond to sysctl settings:
kernel.sched_migration_cost
kernel.sched_autogroup_enabled
What do these settings do?
--------------------------
* sched_migration_cost
The migration cost is the total time the scheduler will consider a
migrated process "cache hot" and thus less likely to be re-migrated. By
default, this is 0.5ms (500000 ns), and as the size of the process table
increases, eventually causes the scheduler to break down. On our
systems, after a smooth degradation with increasing connection count,
system CPU spiked from 20 to 70% sustained and TPS was cut by 5-10x once
we crossed some invisible connection count threshold. For us, that was a
pgbench with 900 or more clients.
The migration cost should be increased, almost universally on server
systems with many processes. This means systems like PostgreSQL or
Apache would benefit from having higher migration costs. We've had good
luck with a setting of 5ms (5000000 ns) instead.
When the breakdown occurs, system CPU (as obtained from sar) increases
from 20% on a heavy pgbench (scale 3500 on a 72GB system) to over 70%,
and %nice/%user is cut by half or more. A higher migration cost
essentially eliminates this artificial throttle.
* sched_autogroup_enabled
This is a relatively new patch which Linus lauded back in late 2010. It
basically groups tasks by TTY so perceived responsiveness is improved.
But on server systems, large daemons like PostgreSQL are going to be
launched from the same pseudo-TTY, and be effectively choked out of CPU
cycles in favor of less important tasks.
The default setting is 1 (enabled) on some platforms. By setting this to
0 (disabled), we saw an outright 30% performance boost on the same
pgbench test. A fully cached scale 3500 database on a 72GB system went
from 67k TPS to 82k TPS with 900 client connections.
Total Benefit
-------------
At higher connections counts, such as systems that can't use pooling or
make extensive use of prepared queries, these can massively affect
performance. At 900 connections, our test systems were at 17k TPS
unaltered, but 85k TPS after these two modifications. Even with this
performance boost, we still had 40% CPU free instead of 0%. In effect,
the logarithmic performance of the new scheduler is returned to normal
under large process tables.
Some systems will have a higher "cracking" point than others. The effect
is amplified when a system is under high memory pressure, hence a lot of
expensive queries on a high number of concurrent connections is the
easiest way to replicate these results.
Admins migrating from older systems (RHEL 5.x) may find this especially
shocking, because the old O(1) scheduler was too "stupid" to have these
advanced features, hence it was impossible to cause this kind of behavior.
There's probably still a little room for improvement here, since 30-40%
CPU is still unclaimed in our larger tests. I'd like to see the total
performance drop (175k ideal TPS at 24-connections) decreased. But these
kernel tweaks are rarely discussed anywhere, it seems. There doesn't
seem to be any consensus on how these (and other) scheduler settings
should be modified under different usage scenarios.
I just figured I'd share, since we found this info so beneficial.
--
Shaun Thomas
OptionsHouse | 141 W. Jackson Blvd. | Suite 500 | Chicago IL, 60604
312-444-8534
sthomas@optionshouse.com
______________________________________________
See http://www.peak6.com/email_disclaimer/ for terms and conditions related to this email
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Two Necessary Kernel Tweaks for Linux Systems
2013-01-02 21:46 Two Necessary Kernel Tweaks for Linux Systems Shaun Thomas <sthomas@optionshouse.com>
@ 2013-01-03 00:47 ` Richard Neill <rn214@richardneill.org>
3 siblings, 0 replies; 17+ messages in thread
From: Richard Neill @ 2013-01-03 00:47 UTC (permalink / raw)
To: sthomas@optionshouse.com; +Cc: pgsql-performance
Dear Shaun,
Thanks for that - it's really interesting to know.
On 02/01/13 21:46, Shaun Thomas wrote:
> Hey everyone!
>
> After much testing and hair-pulling, we've confirmed two kernel
> settings that should always be modified in production Linux systems.
> Especially new ones with the completely fair scheduler (CFS) as
> opposed to the O(1) scheduler.
Does it apply to all types of production system, or just to certain
workloads?
For example, what happens when there are only one or two concurrent
processes? (i.e. there are always several more CPU cores than there are
actual connections).
> * sched_autogroup_enabled
>
> This is a relatively new patch which Linus lauded back in late 2010.
> It basically groups tasks by TTY so perceived responsiveness is
> improved. But on server systems, large daemons like PostgreSQL are
> going to be launched from the same pseudo-TTY, and be effectively
> choked out of CPU cycles in favor of less important tasks.
I've got several production servers using Postgres: I'd like to squeeze
a bit more performance out of them, but in all cases, one (sometimes
two) CPU cores are (sometimes) maxed out, but there are always several
cores permanently idling. So does this apply here?
Thanks for your advice,
Richard
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Two Necessary Kernel Tweaks for Linux Systems
2013-01-02 21:46 Two Necessary Kernel Tweaks for Linux Systems Shaun Thomas <sthomas@optionshouse.com>
@ 2013-01-07 19:22 ` Merlin Moncure <mmoncure@gmail.com>
3 siblings, 0 replies; 17+ messages in thread
From: Merlin Moncure @ 2013-01-07 19:22 UTC (permalink / raw)
To: sthomas@optionshouse.com; +Cc: pgsql-performance; Vlad <marchenko@gmail.com>
On Wed, Jan 2, 2013 at 3:46 PM, Shaun Thomas <sthomas@optionshouse.com> wrote:
> Hey everyone!
>
> After much testing and hair-pulling, we've confirmed two kernel settings
> that should always be modified in production Linux systems. Especially new
> ones with the completely fair scheduler (CFS) as opposed to the O(1)
> scheduler.
>
> If you want to follow along, these are:
>
> /proc/sys/kernel/sched_migration_cost
> /proc/sys/kernel/sched_autogroup_enabled
>
> Which correspond to sysctl settings:
>
> kernel.sched_migration_cost
> kernel.sched_autogroup_enabled
>
> What do these settings do?
> --------------------------
>
> * sched_migration_cost
>
> The migration cost is the total time the scheduler will consider a migrated
> process "cache hot" and thus less likely to be re-migrated. By default, this
> is 0.5ms (500000 ns), and as the size of the process table increases,
> eventually causes the scheduler to break down. On our systems, after a
> smooth degradation with increasing connection count, system CPU spiked from
> 20 to 70% sustained and TPS was cut by 5-10x once we crossed some invisible
> connection count threshold. For us, that was a pgbench with 900 or more
> clients.
>
> The migration cost should be increased, almost universally on server systems
> with many processes. This means systems like PostgreSQL or Apache would
> benefit from having higher migration costs. We've had good luck with a
> setting of 5ms (5000000 ns) instead.
>
> When the breakdown occurs, system CPU (as obtained from sar) increases from
> 20% on a heavy pgbench (scale 3500 on a 72GB system) to over 70%, and
> %nice/%user is cut by half or more. A higher migration cost essentially
> eliminates this artificial throttle.
>
> * sched_autogroup_enabled
>
> This is a relatively new patch which Linus lauded back in late 2010. It
> basically groups tasks by TTY so perceived responsiveness is improved. But
> on server systems, large daemons like PostgreSQL are going to be launched
> from the same pseudo-TTY, and be effectively choked out of CPU cycles in
> favor of less important tasks.
>
> The default setting is 1 (enabled) on some platforms. By setting this to 0
> (disabled), we saw an outright 30% performance boost on the same pgbench
> test. A fully cached scale 3500 database on a 72GB system went from 67k TPS
> to 82k TPS with 900 client connections.
>
> Total Benefit
> -------------
>
> At higher connections counts, such as systems that can't use pooling or make
> extensive use of prepared queries, these can massively affect performance.
> At 900 connections, our test systems were at 17k TPS unaltered, but 85k TPS
> after these two modifications. Even with this performance boost, we still
> had 40% CPU free instead of 0%. In effect, the logarithmic performance of
> the new scheduler is returned to normal under large process tables.
>
> Some systems will have a higher "cracking" point than others. The effect is
> amplified when a system is under high memory pressure, hence a lot of
> expensive queries on a high number of concurrent connections is the easiest
> way to replicate these results.
>
> Admins migrating from older systems (RHEL 5.x) may find this especially
> shocking, because the old O(1) scheduler was too "stupid" to have these
> advanced features, hence it was impossible to cause this kind of behavior.
>
> There's probably still a little room for improvement here, since 30-40% CPU
> is still unclaimed in our larger tests. I'd like to see the total
> performance drop (175k ideal TPS at 24-connections) decreased. But these
> kernel tweaks are rarely discussed anywhere, it seems. There doesn't seem to
> be any consensus on how these (and other) scheduler settings should be
> modified under different usage scenarios.
>
> I just figured I'd share, since we found this info so beneficial.
This is fantastic info.
Vlad, you might want to check this out and see if it has any impact in
your high cpu case...via:
http://postgresql.1045698.n5.nabble.com/High-SYS-CPU-need-advise-td5732045.html
merlin
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Two Necessary Kernel Tweaks for Linux Systems
2013-01-02 21:46 Two Necessary Kernel Tweaks for Linux Systems Shaun Thomas <sthomas@optionshouse.com>
@ 2013-01-08 08:29 ` Andrea Suisani <sickpig@opinioni.net>
2013-01-08 15:16 ` Re: Two Necessary Kernel Tweaks for Linux Systems Andrea Suisani <sickpig@opinioni.net>
3 siblings, 1 reply; 17+ messages in thread
From: Andrea Suisani @ 2013-01-08 08:29 UTC (permalink / raw)
To: sthomas@optionshouse.com; +Cc: pgsql-performance
On 01/02/2013 10:46 PM, Shaun Thomas wrote:
> Hey everyone!
>
> After much testing and hair-pulling, we've confirmed two kernel settings that
> should always be modified in production Linux systems. Especially new ones with
> the completely fair scheduler (CFS) as opposed to the O(1) scheduler.
[cut]
> I just figured I'd share, since we found this info so beneficial.
I just want to confirm that on our relatively small
test server that tweaks give us a 25% performance boost!
Really appreciated Shaun.
thanks
Andrea
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Two Necessary Kernel Tweaks for Linux Systems
2013-01-02 21:46 Two Necessary Kernel Tweaks for Linux Systems Shaun Thomas <sthomas@optionshouse.com>
2013-01-08 08:29 ` Re: Two Necessary Kernel Tweaks for Linux Systems Andrea Suisani <sickpig@opinioni.net>
@ 2013-01-08 15:16 ` Andrea Suisani <sickpig@opinioni.net>
0 siblings, 0 replies; 17+ messages in thread
From: Andrea Suisani @ 2013-01-08 15:16 UTC (permalink / raw)
To: pgsql-performance
On 01/08/2013 09:29 AM, Andrea Suisani wrote:
> On 01/02/2013 10:46 PM, Shaun Thomas wrote:
>> Hey everyone!
>>
>> After much testing and hair-pulling, we've confirmed two kernel settings that
> > should always be modified in production Linux systems. Especially new ones with
>> the completely fair scheduler (CFS) as opposed to the O(1) scheduler.
>
> [cut]
>
>> I just figured I'd share, since we found this info so beneficial.
>
> I just want to confirm that on our relatively small
> test server that tweaks give us a 25% performance boost!
12.5% sorry for the typo...
> Really appreciated Shaun.
>
> thanks
> Andrea
>
>
>
>
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Two Necessary Kernel Tweaks for Linux Systems
2013-01-02 21:46 Two Necessary Kernel Tweaks for Linux Systems Shaun Thomas <sthomas@optionshouse.com>
@ 2013-01-08 18:25 ` Midge Brown <midgems@sbcglobal.net>
2013-01-08 18:28 ` Re: Two Necessary Kernel Tweaks for Linux Systems Shaun Thomas <sthomas@optionshouse.com>
3 siblings, 1 reply; 17+ messages in thread
From: Midge Brown @ 2013-01-08 18:25 UTC (permalink / raw)
To: sthomas@optionshouse.com; pgsql-performance
The kernel on our Linux system doesn't appear to have these two settings according to the list provided by sysctl -a. Please pardon my ignorance, but should I add them?
We have Postgresql 9.0 on Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux
Thanks,
Midge
----- Original Message -----
From: Shaun Thomas
To: pgsql-performance@postgresql.org
Sent: Wednesday, January 02, 2013 1:46 PM
Subject: [PERFORM] Two Necessary Kernel Tweaks for Linux Systems
Hey everyone!
After much testing and hair-pulling, we've confirmed two kernel settings
that should always be modified in production Linux systems. Especially
new ones with the completely fair scheduler (CFS) as opposed to the O(1)
scheduler.
If you want to follow along, these are:
/proc/sys/kernel/sched_migration_cost
/proc/sys/kernel/sched_autogroup_enabled
Which correspond to sysctl settings:
kernel.sched_migration_cost
kernel.sched_autogroup_enabled
What do these settings do?
--------------------------
* sched_migration_cost
The migration cost is the total time the scheduler will consider a
migrated process "cache hot" and thus less likely to be re-migrated. By
default, this is 0.5ms (500000 ns), and as the size of the process table
increases, eventually causes the scheduler to break down. On our
systems, after a smooth degradation with increasing connection count,
system CPU spiked from 20 to 70% sustained and TPS was cut by 5-10x once
we crossed some invisible connection count threshold. For us, that was a
pgbench with 900 or more clients.
The migration cost should be increased, almost universally on server
systems with many processes. This means systems like PostgreSQL or
Apache would benefit from having higher migration costs. We've had good
luck with a setting of 5ms (5000000 ns) instead.
When the breakdown occurs, system CPU (as obtained from sar) increases
from 20% on a heavy pgbench (scale 3500 on a 72GB system) to over 70%,
and %nice/%user is cut by half or more. A higher migration cost
essentially eliminates this artificial throttle.
* sched_autogroup_enabled
This is a relatively new patch which Linus lauded back in late 2010. It
basically groups tasks by TTY so perceived responsiveness is improved.
But on server systems, large daemons like PostgreSQL are going to be
launched from the same pseudo-TTY, and be effectively choked out of CPU
cycles in favor of less important tasks.
The default setting is 1 (enabled) on some platforms. By setting this to
0 (disabled), we saw an outright 30% performance boost on the same
pgbench test. A fully cached scale 3500 database on a 72GB system went
from 67k TPS to 82k TPS with 900 client connections.
Total Benefit
-------------
At higher connections counts, such as systems that can't use pooling or
make extensive use of prepared queries, these can massively affect
performance. At 900 connections, our test systems were at 17k TPS
unaltered, but 85k TPS after these two modifications. Even with this
performance boost, we still had 40% CPU free instead of 0%. In effect,
the logarithmic performance of the new scheduler is returned to normal
under large process tables.
Some systems will have a higher "cracking" point than others. The effect
is amplified when a system is under high memory pressure, hence a lot of
expensive queries on a high number of concurrent connections is the
easiest way to replicate these results.
Admins migrating from older systems (RHEL 5.x) may find this especially
shocking, because the old O(1) scheduler was too "stupid" to have these
advanced features, hence it was impossible to cause this kind of behavior.
There's probably still a little room for improvement here, since 30-40%
CPU is still unclaimed in our larger tests. I'd like to see the total
performance drop (175k ideal TPS at 24-connections) decreased. But these
kernel tweaks are rarely discussed anywhere, it seems. There doesn't
seem to be any consensus on how these (and other) scheduler settings
should be modified under different usage scenarios.
I just figured I'd share, since we found this info so beneficial.
--
Shaun Thomas
OptionsHouse | 141 W. Jackson Blvd. | Suite 500 | Chicago IL, 60604
312-444-8534
sthomas@optionshouse.com
______________________________________________
See http://www.peak6.com/email_disclaimer/ for terms and conditions related to this email
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Two Necessary Kernel Tweaks for Linux Systems
2013-01-02 21:46 Two Necessary Kernel Tweaks for Linux Systems Shaun Thomas <sthomas@optionshouse.com>
2013-01-08 18:25 ` Re: Two Necessary Kernel Tweaks for Linux Systems Midge Brown <midgems@sbcglobal.net>
@ 2013-01-08 18:28 ` Shaun Thomas <sthomas@optionshouse.com>
2013-01-08 18:31 ` Re: Two Necessary Kernel Tweaks for Linux Systems Scott Marlowe <scott.marlowe@gmail.com>
0 siblings, 1 reply; 17+ messages in thread
From: Shaun Thomas @ 2013-01-08 18:28 UTC (permalink / raw)
To: Midge Brown <midgems@sbcglobal.net>; +Cc: pgsql-performance
On 01/08/2013 12:25 PM, Midge Brown wrote:
> The kernel on our Linux system doesn't appear to have these two
> settings according to the list provided by sysctl -a. Please pardon
> my ignorance, but should I add them?
Sorry if I wasn't more clear. These only apply to Linux systems with the
Completely Fair Scheduler, as opposed to the O(1) scheduler. For all
intents and purposes, this means 3.0 kernels and above.
With a 2.6 kernel, you're fine.
Effectively these changes fix what is basically a performance regression
compared to older kernels.
--
Shaun Thomas
OptionsHouse | 141 W. Jackson Blvd. | Suite 500 | Chicago IL, 60604
312-444-8534
sthomas@optionshouse.com
______________________________________________
See http://www.peak6.com/email_disclaimer/ for terms and conditions related to this email
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Two Necessary Kernel Tweaks for Linux Systems
2013-01-02 21:46 Two Necessary Kernel Tweaks for Linux Systems Shaun Thomas <sthomas@optionshouse.com>
2013-01-08 18:25 ` Re: Two Necessary Kernel Tweaks for Linux Systems Midge Brown <midgems@sbcglobal.net>
2013-01-08 18:28 ` Re: Two Necessary Kernel Tweaks for Linux Systems Shaun Thomas <sthomas@optionshouse.com>
@ 2013-01-08 18:31 ` Scott Marlowe <scott.marlowe@gmail.com>
2013-01-08 18:36 ` Re: Two Necessary Kernel Tweaks for Linux Systems Shaun Thomas <sthomas@optionshouse.com>
0 siblings, 1 reply; 17+ messages in thread
From: Scott Marlowe @ 2013-01-08 18:31 UTC (permalink / raw)
To: sthomas@optionshouse.com; +Cc: Midge Brown <midgems@sbcglobal.net>; pgsql-performance
On Tue, Jan 8, 2013 at 11:28 AM, Shaun Thomas <sthomas@optionshouse.com> wrote:
> On 01/08/2013 12:25 PM, Midge Brown wrote:
>
>> The kernel on our Linux system doesn't appear to have these two
>> settings according to the list provided by sysctl -a. Please pardon
>> my ignorance, but should I add them?
>
>
> Sorry if I wasn't more clear. These only apply to Linux systems with the
> Completely Fair Scheduler, as opposed to the O(1) scheduler. For all intents
> and purposes, this means 3.0 kernels and above.
>
> With a 2.6 kernel, you're fine.
>
> Effectively these changes fix what is basically a performance regression
> compared to older kernels.
What's the comparison of these settings versus say going to the NOP scheduler?
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Two Necessary Kernel Tweaks for Linux Systems
2013-01-02 21:46 Two Necessary Kernel Tweaks for Linux Systems Shaun Thomas <sthomas@optionshouse.com>
2013-01-08 18:25 ` Re: Two Necessary Kernel Tweaks for Linux Systems Midge Brown <midgems@sbcglobal.net>
2013-01-08 18:28 ` Re: Two Necessary Kernel Tweaks for Linux Systems Shaun Thomas <sthomas@optionshouse.com>
2013-01-08 18:31 ` Re: Two Necessary Kernel Tweaks for Linux Systems Scott Marlowe <scott.marlowe@gmail.com>
@ 2013-01-08 18:36 ` Shaun Thomas <sthomas@optionshouse.com>
2013-01-08 19:04 ` Re: Two Necessary Kernel Tweaks for Linux Systems Scott Marlowe <scott.marlowe@gmail.com>
0 siblings, 1 reply; 17+ messages in thread
From: Shaun Thomas @ 2013-01-08 18:36 UTC (permalink / raw)
To: Scott Marlowe <scott.marlowe@gmail.com>; +Cc: Midge Brown <midgems@sbcglobal.net>; pgsql-performance
On 01/08/2013 12:31 PM, Scott Marlowe wrote:
> What's the comparison of these settings versus say going to the NOP
> scheduler?
Assuming you actually meant NOP and not the NOOP I/O scheduler, I don't
know. These CPU scheduler tweaks are all I could dig up, and googling
for NOP by itself or combined with Linux terms is tremendously unhelpful.
--
Shaun Thomas
OptionsHouse | 141 W. Jackson Blvd. | Suite 500 | Chicago IL, 60604
312-444-8534
sthomas@optionshouse.com
______________________________________________
See http://www.peak6.com/email_disclaimer/ for terms and conditions related to this email
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Two Necessary Kernel Tweaks for Linux Systems
2013-01-02 21:46 Two Necessary Kernel Tweaks for Linux Systems Shaun Thomas <sthomas@optionshouse.com>
2013-01-08 18:25 ` Re: Two Necessary Kernel Tweaks for Linux Systems Midge Brown <midgems@sbcglobal.net>
2013-01-08 18:28 ` Re: Two Necessary Kernel Tweaks for Linux Systems Shaun Thomas <sthomas@optionshouse.com>
2013-01-08 18:31 ` Re: Two Necessary Kernel Tweaks for Linux Systems Scott Marlowe <scott.marlowe@gmail.com>
2013-01-08 18:36 ` Re: Two Necessary Kernel Tweaks for Linux Systems Shaun Thomas <sthomas@optionshouse.com>
@ 2013-01-08 19:04 ` Scott Marlowe <scott.marlowe@gmail.com>
2013-01-08 19:32 ` Re: Two Necessary Kernel Tweaks for Linux Systems Shaun Thomas <sthomas@optionshouse.com>
0 siblings, 1 reply; 17+ messages in thread
From: Scott Marlowe @ 2013-01-08 19:04 UTC (permalink / raw)
To: sthomas@optionshouse.com; +Cc: Midge Brown <midgems@sbcglobal.net>; pgsql-performance
On Tue, Jan 8, 2013 at 11:36 AM, Shaun Thomas <sthomas@optionshouse.com> wrote:
> On 01/08/2013 12:31 PM, Scott Marlowe wrote:
>
>> What's the comparison of these settings versus say going to the NOP
>> scheduler?
>
>
> Assuming you actually meant NOP and not the NOOP I/O scheduler, I don't
> know. These CPU scheduler tweaks are all I could dig up, and googling for
> NOP by itself or combined with Linux terms is tremendously unhelpful.
Assembly language on the brain. of course I meant NOOP.
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Two Necessary Kernel Tweaks for Linux Systems
2013-01-02 21:46 Two Necessary Kernel Tweaks for Linux Systems Shaun Thomas <sthomas@optionshouse.com>
2013-01-08 18:25 ` Re: Two Necessary Kernel Tweaks for Linux Systems Midge Brown <midgems@sbcglobal.net>
2013-01-08 18:28 ` Re: Two Necessary Kernel Tweaks for Linux Systems Shaun Thomas <sthomas@optionshouse.com>
2013-01-08 18:31 ` Re: Two Necessary Kernel Tweaks for Linux Systems Scott Marlowe <scott.marlowe@gmail.com>
2013-01-08 18:36 ` Re: Two Necessary Kernel Tweaks for Linux Systems Shaun Thomas <sthomas@optionshouse.com>
2013-01-08 19:04 ` Re: Two Necessary Kernel Tweaks for Linux Systems Scott Marlowe <scott.marlowe@gmail.com>
@ 2013-01-08 19:32 ` Shaun Thomas <sthomas@optionshouse.com>
2013-01-08 20:05 ` Re: Two Necessary Kernel Tweaks for Linux Systems AJ Weber <aweber@comcast.net>
2013-01-10 08:51 ` Re: Two Necessary Kernel Tweaks for Linux Systems Henri Philipps <henri.philipps@gmail.com>
0 siblings, 2 replies; 17+ messages in thread
From: Shaun Thomas @ 2013-01-08 19:32 UTC (permalink / raw)
To: Scott Marlowe <scott.marlowe@gmail.com>; +Cc: Midge Brown <midgems@sbcglobal.net>; pgsql-performance
On 01/08/2013 01:04 PM, Scott Marlowe wrote:
> Assembly language on the brain. of course I meant NOOP.
Ok, in that case, these are completely separate things. For IO
scheduling, there's the Completely Fair Queue (CFQ), NOOP, Deadline, and
so on.
For process scheduling, at least recently, there's Completely Fair
Scheduler or nothing. So far as I can tell, there is no alternative
process scheduler. Just as I can't find an alternative memory manager
that I can tell to stop flushing my freaking active file cache due to
phantom memory pressure. ;)
The tweaks I was discussing in this thread effectively do two things:
1. Stop process grouping by TTY.
On servers, this really is a net performance loss. Especially on heavily
forked apps like PG. System % is about 5% lower since the scheduler is
doing less work, but at the cost of less spreading across available
CPUs. Our systems see a 30% performance hit with grouping enabled,
others may see more or less.
2. Less aggressive process scheduling.
The O(log N) scheduler heuristics collapse at high process counts for
some reason, causing the scheduler to spend more and more time planning
CPU assignments until it spirals completely out of control. I've seen
this behavior on 3.0 kernels straight to 3.5, so it looks like an
inherent weakness of CFS. By increasing migration cost, we make the
scheduler do less work less often, so that weird 70+% system CPU spike
vanishes.
My guess is the increased migration cost basically offsets the point at
which the scheduler would freak out. I've tested up to 2000 connections,
and it responds fine, whereas before we were seeing flaky results as
early as 700 connections.
My guess as to why this is? I think it's due to VSZ as perceived by the
scheduler. To swap processes, it also has to preload L2 and L3 cache for
the assigned process. As the number of PG connections increase, all with
their own VSZ/RSS allocations, the scheduler has more thinking to do. At
a point when the sum of VSZ/RSS eclipses the amount of available RAM,
the scheduler loses nearly all decision-making ability and craps its pants.
This would also explain why I'm seeing something similar with memory. At
high connection counts, even though %used is fine, and we have over 40GB
free for caching. VSZ/RSS are both way bigger than available cache, so
memory pressure causes kswapd to continuously purge the active cache
pool into inactive, and inactive into free, all while the device
attempts to fill the active pool. It's an IO feedback loop, and around
the same number of connections that used to make the process scheduler
die. Too much of a coincidence, in my opinion.
But unlike the process scheduler, there are no good knobs to turn that
will fix the memory manager's behavior. At least, not in 3.0, 3.2, or
3.4 kernels.
But I freely admit I'm just speculating based on observed behavior. I
know neither jack, nor squat about internal kernel mechanics. Anyone who
actually *isn't* talking out of his ass is free to interject. :)
--
Shaun Thomas
OptionsHouse | 141 W. Jackson Blvd. | Suite 500 | Chicago IL, 60604
312-444-8534
sthomas@optionshouse.com
______________________________________________
See http://www.peak6.com/email_disclaimer/ for terms and conditions related to this email
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Two Necessary Kernel Tweaks for Linux Systems
2013-01-02 21:46 Two Necessary Kernel Tweaks for Linux Systems Shaun Thomas <sthomas@optionshouse.com>
2013-01-08 18:25 ` Re: Two Necessary Kernel Tweaks for Linux Systems Midge Brown <midgems@sbcglobal.net>
2013-01-08 18:28 ` Re: Two Necessary Kernel Tweaks for Linux Systems Shaun Thomas <sthomas@optionshouse.com>
2013-01-08 18:31 ` Re: Two Necessary Kernel Tweaks for Linux Systems Scott Marlowe <scott.marlowe@gmail.com>
2013-01-08 18:36 ` Re: Two Necessary Kernel Tweaks for Linux Systems Shaun Thomas <sthomas@optionshouse.com>
2013-01-08 19:04 ` Re: Two Necessary Kernel Tweaks for Linux Systems Scott Marlowe <scott.marlowe@gmail.com>
2013-01-08 19:32 ` Re: Two Necessary Kernel Tweaks for Linux Systems Shaun Thomas <sthomas@optionshouse.com>
@ 2013-01-08 20:05 ` AJ Weber <aweber@comcast.net>
2013-01-08 21:48 ` Re: Two Necessary Kernel Tweaks for Linux Systems Shaun Thomas <sthomas@optionshouse.com>
1 sibling, 1 reply; 17+ messages in thread
From: AJ Weber @ 2013-01-08 20:05 UTC (permalink / raw)
To: pgsql-performance
When I checked these, both of these settings exist on my CentOS 6.x host
(2.6.32-279.5.1.el6.x86_64).
However, the autogroup_enabled was already set to 0. (The
migration_cost was set to the 0.5ms, default noted in the OP.) So I
don't know if this is strictly limited to kernel 3.0.
Is there an "easy" way to tell what scheduler my OS is using?
-AJ
On 1/8/2013 2:32 PM, Shaun Thomas wrote:
> On 01/08/2013 01:04 PM, Scott Marlowe wrote:
>
>> Assembly language on the brain. of course I meant NOOP.
>
> Ok, in that case, these are completely separate things. For IO
> scheduling, there's the Completely Fair Queue (CFQ), NOOP, Deadline,
> and so on.
>
> For process scheduling, at least recently, there's Completely Fair
> Scheduler or nothing. So far as I can tell, there is no alternative
> process scheduler. Just as I can't find an alternative memory manager
> that I can tell to stop flushing my freaking active file cache due to
> phantom memory pressure. ;)
>
> The tweaks I was discussing in this thread effectively do two things:
>
> 1. Stop process grouping by TTY.
>
> On servers, this really is a net performance loss. Especially on
> heavily forked apps like PG. System % is about 5% lower since the
> scheduler is doing less work, but at the cost of less spreading across
> available CPUs. Our systems see a 30% performance hit with grouping
> enabled, others may see more or less.
>
> 2. Less aggressive process scheduling.
>
> The O(log N) scheduler heuristics collapse at high process counts for
> some reason, causing the scheduler to spend more and more time
> planning CPU assignments until it spirals completely out of control.
> I've seen this behavior on 3.0 kernels straight to 3.5, so it looks
> like an inherent weakness of CFS. By increasing migration cost, we
> make the scheduler do less work less often, so that weird 70+% system
> CPU spike vanishes.
>
> My guess is the increased migration cost basically offsets the point
> at which the scheduler would freak out. I've tested up to 2000
> connections, and it responds fine, whereas before we were seeing flaky
> results as early as 700 connections.
>
> My guess as to why this is? I think it's due to VSZ as perceived by
> the scheduler. To swap processes, it also has to preload L2 and L3
> cache for the assigned process. As the number of PG connections
> increase, all with their own VSZ/RSS allocations, the scheduler has
> more thinking to do. At a point when the sum of VSZ/RSS eclipses the
> amount of available RAM, the scheduler loses nearly all
> decision-making ability and craps its pants.
>
> This would also explain why I'm seeing something similar with memory.
> At high connection counts, even though %used is fine, and we have over
> 40GB free for caching. VSZ/RSS are both way bigger than available
> cache, so memory pressure causes kswapd to continuously purge the
> active cache pool into inactive, and inactive into free, all while the
> device attempts to fill the active pool. It's an IO feedback loop, and
> around the same number of connections that used to make the process
> scheduler die. Too much of a coincidence, in my opinion.
>
> But unlike the process scheduler, there are no good knobs to turn that
> will fix the memory manager's behavior. At least, not in 3.0, 3.2, or
> 3.4 kernels.
>
> But I freely admit I'm just speculating based on observed behavior. I
> know neither jack, nor squat about internal kernel mechanics. Anyone
> who actually *isn't* talking out of his ass is free to interject. :)
>
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Two Necessary Kernel Tweaks for Linux Systems
2013-01-02 21:46 Two Necessary Kernel Tweaks for Linux Systems Shaun Thomas <sthomas@optionshouse.com>
2013-01-08 18:25 ` Re: Two Necessary Kernel Tweaks for Linux Systems Midge Brown <midgems@sbcglobal.net>
2013-01-08 18:28 ` Re: Two Necessary Kernel Tweaks for Linux Systems Shaun Thomas <sthomas@optionshouse.com>
2013-01-08 18:31 ` Re: Two Necessary Kernel Tweaks for Linux Systems Scott Marlowe <scott.marlowe@gmail.com>
2013-01-08 18:36 ` Re: Two Necessary Kernel Tweaks for Linux Systems Shaun Thomas <sthomas@optionshouse.com>
2013-01-08 19:04 ` Re: Two Necessary Kernel Tweaks for Linux Systems Scott Marlowe <scott.marlowe@gmail.com>
2013-01-08 19:32 ` Re: Two Necessary Kernel Tweaks for Linux Systems Shaun Thomas <sthomas@optionshouse.com>
2013-01-08 20:05 ` Re: Two Necessary Kernel Tweaks for Linux Systems AJ Weber <aweber@comcast.net>
@ 2013-01-08 21:48 ` Shaun Thomas <sthomas@optionshouse.com>
2013-01-08 23:24 ` Re: Two Necessary Kernel Tweaks for Linux Systems Alan Hodgson <ahodgson@simkin.ca>
2013-01-14 14:28 ` Re: Two Necessary Kernel Tweaks for Linux Systems Boszormenyi Zoltan <zb@cybertec.at>
0 siblings, 2 replies; 17+ messages in thread
From: Shaun Thomas @ 2013-01-08 21:48 UTC (permalink / raw)
To: AJ Weber <aweber@comcast.net>; +Cc: pgsql-performance
On 01/08/2013 02:05 PM, AJ Weber wrote:
> Is there an "easy" way to tell what scheduler my OS is using?
Unfortunately not. I looked again, and it seems that CFS was merged into
2.6.23. Anything before that is probably safe, but the vendor may have
backported it. If you don't see the settings I described, you probably
don't have it.
So I guess Midge had 2.6.18, which predates the merge in 2.6.23.
I honestly don't understand the Linux kernel sometimes. A process
scheduler swap is a *gigantic* functional change, and it's in a dot
release. I vastly prefer PostgreSQL's approach...
--
Shaun Thomas
OptionsHouse | 141 W. Jackson Blvd. | Suite 500 | Chicago IL, 60604
312-444-8534
sthomas@optionshouse.com
______________________________________________
See http://www.peak6.com/email_disclaimer/ for terms and conditions related to this email
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Two Necessary Kernel Tweaks for Linux Systems
2013-01-02 21:46 Two Necessary Kernel Tweaks for Linux Systems Shaun Thomas <sthomas@optionshouse.com>
2013-01-08 18:25 ` Re: Two Necessary Kernel Tweaks for Linux Systems Midge Brown <midgems@sbcglobal.net>
2013-01-08 18:28 ` Re: Two Necessary Kernel Tweaks for Linux Systems Shaun Thomas <sthomas@optionshouse.com>
2013-01-08 18:31 ` Re: Two Necessary Kernel Tweaks for Linux Systems Scott Marlowe <scott.marlowe@gmail.com>
2013-01-08 18:36 ` Re: Two Necessary Kernel Tweaks for Linux Systems Shaun Thomas <sthomas@optionshouse.com>
2013-01-08 19:04 ` Re: Two Necessary Kernel Tweaks for Linux Systems Scott Marlowe <scott.marlowe@gmail.com>
2013-01-08 19:32 ` Re: Two Necessary Kernel Tweaks for Linux Systems Shaun Thomas <sthomas@optionshouse.com>
2013-01-08 20:05 ` Re: Two Necessary Kernel Tweaks for Linux Systems AJ Weber <aweber@comcast.net>
2013-01-08 21:48 ` Re: Two Necessary Kernel Tweaks for Linux Systems Shaun Thomas <sthomas@optionshouse.com>
@ 2013-01-08 23:24 ` Alan Hodgson <ahodgson@simkin.ca>
1 sibling, 0 replies; 17+ messages in thread
From: Alan Hodgson @ 2013-01-08 23:24 UTC (permalink / raw)
To: pgsql-performance
On Tuesday, January 08, 2013 03:48:38 PM Shaun Thomas wrote:
> On 01/08/2013 02:05 PM, AJ Weber wrote:
> > Is there an "easy" way to tell what scheduler my OS is using?
>
> Unfortunately not. I looked again, and it seems that CFS was merged into
> 2.6.23. Anything before that is probably safe, but the vendor may have
> backported it. If you don't see the settings I described, you probably
> don't have it.
>
> So I guess Midge had 2.6.18, which predates the merge in 2.6.23.
>
> I honestly don't understand the Linux kernel sometimes. A process
> scheduler swap is a *gigantic* functional change, and it's in a dot
> release. I vastly prefer PostgreSQL's approach...
Red Hat also selectively backports major functionality into their enterprise
kernels. If you're running RHEL or a clone like CentOS, the reported kernel
version has little bearing on what may nor may not be in your kernel.
They're very well tested and stable, so there's nothing wrong with them, per
se, but you can't just say oh, you have version xxx, you don't have this
functionality.
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Two Necessary Kernel Tweaks for Linux Systems
2013-01-02 21:46 Two Necessary Kernel Tweaks for Linux Systems Shaun Thomas <sthomas@optionshouse.com>
2013-01-08 18:25 ` Re: Two Necessary Kernel Tweaks for Linux Systems Midge Brown <midgems@sbcglobal.net>
2013-01-08 18:28 ` Re: Two Necessary Kernel Tweaks for Linux Systems Shaun Thomas <sthomas@optionshouse.com>
2013-01-08 18:31 ` Re: Two Necessary Kernel Tweaks for Linux Systems Scott Marlowe <scott.marlowe@gmail.com>
2013-01-08 18:36 ` Re: Two Necessary Kernel Tweaks for Linux Systems Shaun Thomas <sthomas@optionshouse.com>
2013-01-08 19:04 ` Re: Two Necessary Kernel Tweaks for Linux Systems Scott Marlowe <scott.marlowe@gmail.com>
2013-01-08 19:32 ` Re: Two Necessary Kernel Tweaks for Linux Systems Shaun Thomas <sthomas@optionshouse.com>
2013-01-08 20:05 ` Re: Two Necessary Kernel Tweaks for Linux Systems AJ Weber <aweber@comcast.net>
2013-01-08 21:48 ` Re: Two Necessary Kernel Tweaks for Linux Systems Shaun Thomas <sthomas@optionshouse.com>
@ 2013-01-14 14:28 ` Boszormenyi Zoltan <zb@cybertec.at>
1 sibling, 0 replies; 17+ messages in thread
From: Boszormenyi Zoltan @ 2013-01-14 14:28 UTC (permalink / raw)
To: sthomas@optionshouse.com; +Cc: AJ Weber <aweber@comcast.net>; pgsql-performance
2013-01-08 22:48 keltezéssel, Shaun Thomas írta:
> On 01/08/2013 02:05 PM, AJ Weber wrote:
>
>> Is there an "easy" way to tell what scheduler my OS is using?
>
> Unfortunately not. I looked again, and it seems that CFS was merged into 2.6.23.
> Anything before that is probably safe, but the vendor may have backported it. If you
> don't see the settings I described, you probably don't have it.
>
> So I guess Midge had 2.6.18, which predates the merge in 2.6.23.
>
> I honestly don't understand the Linux kernel sometimes. A process scheduler swap is a
> *gigantic* functional change, and it's in a dot release. I vastly prefer PostgreSQL's
> approach...
The kernel version numbering is different.
A point release in 2.6.x is 2.6.x.y.
This has changed in 3.x, a point release is 3.x.y.
Best regards,
Zoltán Böszörményi
--
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
http://www.postgresql.at/
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Two Necessary Kernel Tweaks for Linux Systems
2013-01-02 21:46 Two Necessary Kernel Tweaks for Linux Systems Shaun Thomas <sthomas@optionshouse.com>
2013-01-08 18:25 ` Re: Two Necessary Kernel Tweaks for Linux Systems Midge Brown <midgems@sbcglobal.net>
2013-01-08 18:28 ` Re: Two Necessary Kernel Tweaks for Linux Systems Shaun Thomas <sthomas@optionshouse.com>
2013-01-08 18:31 ` Re: Two Necessary Kernel Tweaks for Linux Systems Scott Marlowe <scott.marlowe@gmail.com>
2013-01-08 18:36 ` Re: Two Necessary Kernel Tweaks for Linux Systems Shaun Thomas <sthomas@optionshouse.com>
2013-01-08 19:04 ` Re: Two Necessary Kernel Tweaks for Linux Systems Scott Marlowe <scott.marlowe@gmail.com>
2013-01-08 19:32 ` Re: Two Necessary Kernel Tweaks for Linux Systems Shaun Thomas <sthomas@optionshouse.com>
@ 2013-01-10 08:51 ` Henri Philipps <henri.philipps@gmail.com>
2013-01-10 15:53 ` Re: Two Necessary Kernel Tweaks for Linux Systems Shaun Thomas <sthomas@optionshouse.com>
1 sibling, 1 reply; 17+ messages in thread
From: Henri Philipps @ 2013-01-10 08:51 UTC (permalink / raw)
To: pgsql-performance
Hi,
we also hit this performance barrier a while ago, when migrating a
database on a big server (48 core Opteron, 512GB RAM) from Kernel
2.6.32 to 3.2 (both kernels from Debian packages). The system load was
getting very high, as you also observed (don't know the exact numbers
right now).
After some investigation I found out, that the reason for the high
system load was that the postgresql processes were migrating from core
to core at very high rates. So the behaviour of the CFS scheduler must
have changed in this regard between 2.6.32 and 3.2 kernels.
You can easily see this, if you have a look how much time the
migration kernel threads spend in the CPU (ps ax | grep migration). A
look into /proc/sched_debug also can give you some more insight into
the scheduler behaviour.
On NUMA systems the scheduler tries to migrate processes to the nodes
on which they have the best memory-locality. But on a big database one
process is typically reading randomly from a dataset which is spread
above all nodes. On newer kernels the CFS scheduler seems to try more
aggressively to migrate processes to other cores. I don't know if it
is for better load balancing or for better memory locality. But
process migrations are consuming a lot of resources.
I had to change sched_migration_costs from 500000 (0.5ms) to 100000000
(100ms). This means, the scheduler is only considering a task for
migration if the task was running at least for 100ms instead of 0.5ms.
This solved the problem for us - the migration kernel threads didn't
have to do much work anymore and thus the system load was going down
again.
A general problem is, that the CFS scheduler has a lot of changes
between all kernel versions, so it is really hard to predict which
regressions you can hit when going to another kernel version.
Scheduling on NUMA systems is also very complex.
An interesting dissertations showing the inconsistent behaviour of the
CFS scheduler:
http://research.cs.wisc.edu/adsl/Publications/meehean-thesis11.pdf
Some parameters, which also could be considered for systematic benchmarking are
sched_latency_ns
sched_min_granularity_ns
I guess that higher numbers could improve performance too on systems
with many cores and many connections.
Thanks for starting this interesting thread!
Henri
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Two Necessary Kernel Tweaks for Linux Systems
2013-01-02 21:46 Two Necessary Kernel Tweaks for Linux Systems Shaun Thomas <sthomas@optionshouse.com>
2013-01-08 18:25 ` Re: Two Necessary Kernel Tweaks for Linux Systems Midge Brown <midgems@sbcglobal.net>
2013-01-08 18:28 ` Re: Two Necessary Kernel Tweaks for Linux Systems Shaun Thomas <sthomas@optionshouse.com>
2013-01-08 18:31 ` Re: Two Necessary Kernel Tweaks for Linux Systems Scott Marlowe <scott.marlowe@gmail.com>
2013-01-08 18:36 ` Re: Two Necessary Kernel Tweaks for Linux Systems Shaun Thomas <sthomas@optionshouse.com>
2013-01-08 19:04 ` Re: Two Necessary Kernel Tweaks for Linux Systems Scott Marlowe <scott.marlowe@gmail.com>
2013-01-08 19:32 ` Re: Two Necessary Kernel Tweaks for Linux Systems Shaun Thomas <sthomas@optionshouse.com>
2013-01-10 08:51 ` Re: Two Necessary Kernel Tweaks for Linux Systems Henri Philipps <henri.philipps@gmail.com>
@ 2013-01-10 15:53 ` Shaun Thomas <sthomas@optionshouse.com>
0 siblings, 0 replies; 17+ messages in thread
From: Shaun Thomas @ 2013-01-10 15:53 UTC (permalink / raw)
To: Henri Philipps <henri.philipps@gmail.com>; +Cc: pgsql-performance
On 01/10/2013 02:51 AM, Henri Philipps wrote:
> http://research.cs.wisc.edu/adsl/Publications/meehean-thesis11.pdf
Wow, that was pretty interesting. It looks like for servers, the O(1)
scheduler is much better even with the assignment bug he identified, and
BFS responds better to varying load than CFS.
It's too bad the paper is so old and only considers the 2.6 kernel. I'd
love to see this type of research applied to the latest.
> sched_latency_ns
> sched_min_granularity_ns
>
> I guess that higher numbers could improve performance too on systems
> with many cores and many connections.
I messed around with these a bit. Settings 10x smaller and 10x larger
didn't do anything appreciable that I noticed. Performance metrics were
within variance of my earlier tests. Only autogrouping and migration
cost had any appreciable effect.
I'm glad we weren't the only ones who ran into this, too. You settled on
a much higher setting than we did, but the end result was the same. I
wonder how prevalent this will become as more servers are switched over
to newer kernels in the next couple of years. Hopefully more people
start complaining so they fix it. :)
--
Shaun Thomas
OptionsHouse | 141 W. Jackson Blvd. | Suite 500 | Chicago IL, 60604
312-676-8870
sthomas@optionshouse.com
______________________________________________
See http://www.peak6.com/email_disclaimer/ for terms and conditions related to this email
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 17+ messages in thread
end of thread, other threads:[~2013-01-14 14:28 UTC | newest]
Thread overview: 17+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2013-01-02 21:46 Two Necessary Kernel Tweaks for Linux Systems Shaun Thomas <sthomas@optionshouse.com>
2013-01-03 00:47 ` Richard Neill <rn214@richardneill.org>
2013-01-07 19:22 ` Merlin Moncure <mmoncure@gmail.com>
2013-01-08 08:29 ` Andrea Suisani <sickpig@opinioni.net>
2013-01-08 15:16 ` Andrea Suisani <sickpig@opinioni.net>
2013-01-08 18:25 ` Midge Brown <midgems@sbcglobal.net>
2013-01-08 18:28 ` Shaun Thomas <sthomas@optionshouse.com>
2013-01-08 18:31 ` Scott Marlowe <scott.marlowe@gmail.com>
2013-01-08 18:36 ` Shaun Thomas <sthomas@optionshouse.com>
2013-01-08 19:04 ` Scott Marlowe <scott.marlowe@gmail.com>
2013-01-08 19:32 ` Shaun Thomas <sthomas@optionshouse.com>
2013-01-08 20:05 ` AJ Weber <aweber@comcast.net>
2013-01-08 21:48 ` Shaun Thomas <sthomas@optionshouse.com>
2013-01-08 23:24 ` Alan Hodgson <ahodgson@simkin.ca>
2013-01-14 14:28 ` Boszormenyi Zoltan <zb@cybertec.at>
2013-01-10 08:51 ` Henri Philipps <henri.philipps@gmail.com>
2013-01-10 15:53 ` Shaun Thomas <sthomas@optionshouse.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox