Message-ID: <50DCA815.9070302@optionshouse.com>
Date: Thu, 27 Dec 2012 13:57:09 -0600
From: Shaun Thomas <sthomas@optionshouse.com>
Reply-To: <sthomas@optionshouse.com>
Organization: Peak6
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
	rv:17.0) Gecko/17.0 Thunderbird/17.0
MIME-Version: 1.0
To: "pgsql-performance@postgresql.org" <pgsql-performance@postgresql.org>
Subject: sched_migration_cost for high-connection counts
Content-Type: text/plain; charset="ISO-8859-1"; format=flowed
content-transfer-encoding: 7bit
Precedence: bulk
Sender: pgsql-performance-owner@postgresql.org

Hey guys,

I recently stumbled over a Linux scheduler setting that has outright 
shocked me. So, after reading through this:

http://blog.tsunanet.net/2010/11/how-long-does-it-take-to-make-context.html

it became readily apparent we were hitting the same wall. I could do a 
pgbench and increase the connection count by 100 every iteration, and 
eventually performance just fell off a proverbial cliff and never recovered.

For our particular systems, this barrier is somewhere around 800 
processes. Select-only performance on a 3600-scale pgbench database in 
cache falls from about 70k TPS to about 12k TPS after crossing that 
line. Worse, sar shows over 70% CPU dedicated to system overhead.

After some fiddling around, I changed sched_migration_cost from its 
default of 500000 to 5000000 and performance returned to linear scaling 
immediately. It's literally night and day. Setting it back to 500000 
reverts to the terrible performance.

In addition, setting the migration cost to a higher value does not 
negatively affect any other performance metric I've checked. This is on 
an Ubuntu 12.04 system, and I'd love if someone out there could 
independently verify this, because frankly, I find it difficult to believe.

If legit, high-connection systems would benefit greatly.

-- 
Shaun Thomas
OptionsHouse | 141 W. Jackson Blvd. | Suite 500 | Chicago IL, 60604
312-444-8534
sthomas@optionshouse.com

______________________________________________

See http://www.peak6.com/email_disclaimer/ for terms and conditions related to this email


-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance