Message-ID: <53BFA63C.60909@catalyst.net.nz>
Date: Fri, 11 Jul 2014 20:54:20 +1200
From: Mark Kirkwood <mark.kirkwood@catalyst.net.nz>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:24.0) Gecko/20100101 Thunderbird/24.6.0
MIME-Version: 1.0
To: Andres Freund <andres@2ndquadrant.com>
CC: pgsql-performance@postgresql.org
Subject: Re: 60 core performance with 9.3
References: <53ACB190.2030206@catalyst.net.nz>
 <CAOR=d=3=Y6yNtwQenwVBGe6ZgjAeA8PwgYb3=aQEKn+vYWhBJA@mail.gmail.com>
 <53ACD6C4.9050605@catalyst.net.nz>
 <20140627091951.GA16680@awork2.anarazel.de>
 <53B283F3.7020005@catalyst.net.nz>
 <20140701101307.GO21422@awork2.anarazel.de>
 <53BF326F.2070903@catalyst.net.nz>
 <20140711082207.GI17261@awork2.anarazel.de>
In-Reply-To: <20140711082207.GI17261@awork2.anarazel.de>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
Sender: pgsql-performance-owner@postgresql.org

On 11/07/14 20:22, Andres Freund wrote:
> On 2014-07-11 12:40:15 +1200, Mark Kirkwood wrote:

>> Postgres 9.4 beta
>> rwlock patch
>> pgbench scale = 2000
>>
> On that scale - that's bigger than shared_buffers IIRC - I'd not expect
> the patch to make much of a difference.
>

Right - we did test with it bigger (can't recall exactly how big), but 
will retry again after setting the numa parameters below.

>> #
>>       8.82%        postgres  [kernel.kallsyms]        [k]
>> _raw_spin_lock_irqsave
>>                    |
>>                    --- _raw_spin_lock_irqsave
>>                       |
>>                       |--75.69%-- pagevec_lru_move_fn
>>                       |          __lru_cache_add
>>                       |          lru_cache_add
>>                       |          putback_lru_page
>>                       |          migrate_pages
>>                       |          migrate_misplaced_page
>>                       |          do_numa_page
>>                       |          handle_mm_fault
>>                       |          __do_page_fault
>>                       |          do_page_fault
>>                       |          page_fault
>
> So, the majority of the time is spent in numa page migration. Can you
> disable numa_balancing? I'm not sure if your kernel version does that at
> runtime or whether you need to reboot.
> The kernel.numa_balancing sysctl might work. Otherwise you probably need
> to boot with numa_balancing=0.
>
> It'd also be worthwhile to test this with numactl --interleave.
>

That was my feeling too - but I had no idea what the magic switch was to 
tame it (appears to be in 3.13 kernels), will experiment and report 
back. Thanks again!

Mark


-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance