From: AJ Weber <aweber@comcast.net>
To: pgsql-performance@postgresql.org
Subject: Re: Two Necessary Kernel Tweaks for Linux Systems
Date: Tue, 08 Jan 2013 15:05:56 -0500
Message-ID: <50EC7C24.2020203@comcast.net> (raw)
In-Reply-To: <50EC743E.1040209@optionshouse.com>
References: <50E4AAB1.9040902@optionshouse.com>
<B994348460014EC7BA65F310E9D30EBD@BERNICE>
<50EC6549.3090503@optionshouse.com>
<CAOR=d=2g4iH+4U2Dkn2EZtTnZrescxp+MW10zehVUzX6a5GXSA@mail.gmail.com>
<50EC6742.4090303@optionshouse.com>
<CAOR=d=0R1GPVfhhy9w2+j=czNvgtbA519rY6-Or6sNegvQmdFQ@mail.gmail.com>
<50EC743E.1040209@optionshouse.com>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-performance>
When I checked these, both of these settings exist on my CentOS 6.x host
(2.6.32-279.5.1.el6.x86_64).
However, the autogroup_enabled was already set to 0. (The
migration_cost was set to the 0.5ms, default noted in the OP.) So I
don't know if this is strictly limited to kernel 3.0.
Is there an "easy" way to tell what scheduler my OS is using?
-AJ
On 1/8/2013 2:32 PM, Shaun Thomas wrote:
> On 01/08/2013 01:04 PM, Scott Marlowe wrote:
>
>> Assembly language on the brain. of course I meant NOOP.
>
> Ok, in that case, these are completely separate things. For IO
> scheduling, there's the Completely Fair Queue (CFQ), NOOP, Deadline,
> and so on.
>
> For process scheduling, at least recently, there's Completely Fair
> Scheduler or nothing. So far as I can tell, there is no alternative
> process scheduler. Just as I can't find an alternative memory manager
> that I can tell to stop flushing my freaking active file cache due to
> phantom memory pressure. ;)
>
> The tweaks I was discussing in this thread effectively do two things:
>
> 1. Stop process grouping by TTY.
>
> On servers, this really is a net performance loss. Especially on
> heavily forked apps like PG. System % is about 5% lower since the
> scheduler is doing less work, but at the cost of less spreading across
> available CPUs. Our systems see a 30% performance hit with grouping
> enabled, others may see more or less.
>
> 2. Less aggressive process scheduling.
>
> The O(log N) scheduler heuristics collapse at high process counts for
> some reason, causing the scheduler to spend more and more time
> planning CPU assignments until it spirals completely out of control.
> I've seen this behavior on 3.0 kernels straight to 3.5, so it looks
> like an inherent weakness of CFS. By increasing migration cost, we
> make the scheduler do less work less often, so that weird 70+% system
> CPU spike vanishes.
>
> My guess is the increased migration cost basically offsets the point
> at which the scheduler would freak out. I've tested up to 2000
> connections, and it responds fine, whereas before we were seeing flaky
> results as early as 700 connections.
>
> My guess as to why this is? I think it's due to VSZ as perceived by
> the scheduler. To swap processes, it also has to preload L2 and L3
> cache for the assigned process. As the number of PG connections
> increase, all with their own VSZ/RSS allocations, the scheduler has
> more thinking to do. At a point when the sum of VSZ/RSS eclipses the
> amount of available RAM, the scheduler loses nearly all
> decision-making ability and craps its pants.
>
> This would also explain why I'm seeing something similar with memory.
> At high connection counts, even though %used is fine, and we have over
> 40GB free for caching. VSZ/RSS are both way bigger than available
> cache, so memory pressure causes kswapd to continuously purge the
> active cache pool into inactive, and inactive into free, all while the
> device attempts to fill the active pool. It's an IO feedback loop, and
> around the same number of connections that used to make the process
> scheduler die. Too much of a coincidence, in my opinion.
>
> But unlike the process scheduler, there are no good knobs to turn that
> will fix the memory manager's behavior. At least, not in 3.0, 3.2, or
> 3.4 kernels.
>
> But I freely admit I'm just speculating based on observed behavior. I
> know neither jack, nor squat about internal kernel mechanics. Anyone
> who actually *isn't* talking out of his ass is free to interject. :)
>
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: aweber@comcast.net
Subject: Re: Two Necessary Kernel Tweaks for Linux Systems
In-Reply-To: <50EC7C24.2020203@comcast.net>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox