From: Shaun Thomas <sthomas@optionshouse.com>
To: pgsql-performance@postgresql.org <pgsql-performance@postgresql.org>
Subject: sched_migration_cost for high-connection counts
Date: Thu, 27 Dec 2012 13:57:09 -0600
Message-ID: <50DCA815.9070302@optionshouse.com> (raw)
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-performance>
Hey guys,
I recently stumbled over a Linux scheduler setting that has outright
shocked me. So, after reading through this:
http://blog.tsunanet.net/2010/11/how-long-does-it-take-to-make-context.html
it became readily apparent we were hitting the same wall. I could do a
pgbench and increase the connection count by 100 every iteration, and
eventually performance just fell off a proverbial cliff and never recovered.
For our particular systems, this barrier is somewhere around 800
processes. Select-only performance on a 3600-scale pgbench database in
cache falls from about 70k TPS to about 12k TPS after crossing that
line. Worse, sar shows over 70% CPU dedicated to system overhead.
After some fiddling around, I changed sched_migration_cost from its
default of 500000 to 5000000 and performance returned to linear scaling
immediately. It's literally night and day. Setting it back to 500000
reverts to the terrible performance.
In addition, setting the migration cost to a higher value does not
negatively affect any other performance metric I've checked. This is on
an Ubuntu 12.04 system, and I'd love if someone out there could
independently verify this, because frankly, I find it difficult to believe.
If legit, high-connection systems would benefit greatly.
--
Shaun Thomas
OptionsHouse | 141 W. Jackson Blvd. | Suite 500 | Chicago IL, 60604
312-444-8534
sthomas@optionshouse.com
______________________________________________
See http://www.peak6.com/email_disclaimer/ for terms and conditions related to this email
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: sthomas@optionshouse.com
Subject: Re: sched_migration_cost for high-connection counts
In-Reply-To: <50DCA815.9070302@optionshouse.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox