Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtp (Exim 4.80) (envelope-from ) id 1X7Z5m-0004bO-9k for pgsql-performance@arkaria.postgresql.org; Wed, 16 Jul 2014 23:58:34 +0000 Received: from localhost ([127.0.0.1] helo=postgresql.org) by malur.postgresql.org with smtp (Exim 4.80) (envelope-from ) id 1X7Z5l-0006PM-LI for pgsql-performance@arkaria.postgresql.org; Wed, 16 Jul 2014 23:58:33 +0000 Received: from makus.postgresql.org ([2001:4800:1501:1::229]) by malur.postgresql.org with esmtps (TLS1.2:DHE_RSA_AES_256_CBC_SHA256:256) (Exim 4.80) (envelope-from ) id 1X7Z5j-0006PD-Ek for pgsql-performance@postgresql.org; Wed, 16 Jul 2014 23:58:31 +0000 Received: from [2404:130:0:10::40:2] (helo=mail.catalyst.net.nz) by makus.postgresql.org with esmtp (Exim 4.80) (envelope-from ) id 1X7Z5e-0006Ro-FK for pgsql-performance@postgresql.org; Wed, 16 Jul 2014 23:58:29 +0000 Received: from localhost (localhost [127.0.0.1]) by mail.catalyst.net.nz (Postfix) with ESMTP id 82154323C2; Thu, 17 Jul 2014 11:58:20 +1200 (NZST) X-Virus-Scanned: Debian amavisd-new at catalyst.net.nz Received: from mail.catalyst.net.nz ([127.0.0.1]) by localhost (bertrand.catalyst.net.nz [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id thehXO+ZBHhj; Thu, 17 Jul 2014 11:58:10 +1200 (NZST) Received: from [IPv6:2404:130:0:1000:e52f:55a:209e:3fce] (unknown [IPv6:2404:130:0:1000:e52f:55a:209e:3fce]) (Authenticated sender: mark.kirkwood) by mail.catalyst.net.nz (Postfix) with ESMTPSA id 00ACB327EC; Thu, 17 Jul 2014 11:54:25 +1200 (NZST) Message-ID: <53C711B6.5040802@catalyst.net.nz> Date: Thu, 17 Jul 2014 11:58:46 +1200 From: Mark Kirkwood User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: Andres Freund CC: pgsql-performance@postgresql.org Subject: Re: 60 core performance with 9.3 References: <53ACB190.2030206@catalyst.net.nz> <53ACD6C4.9050605@catalyst.net.nz> <20140627091951.GA16680@awork2.anarazel.de> <53B283F3.7020005@catalyst.net.nz> <20140701101307.GO21422@awork2.anarazel.de> <53BF326F.2070903@catalyst.net.nz> <20140711082207.GI17261@awork2.anarazel.de> In-Reply-To: <20140711082207.GI17261@awork2.anarazel.de> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Host-Lookup-Failed: Reverse DNS lookup failed for 2404:130:0:10::40:2 (failed) X-Pg-Spam-Score: 0.8 (/) List-Archive: List-Help: List-ID: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: X-Mailing-List: pgsql-performance Precedence: bulk Sender: pgsql-performance-owner@postgresql.org On 11/07/14 20:22, Andres Freund wrote: > On 2014-07-11 12:40:15 +1200, Mark Kirkwood wrote: >> Full report http://paste.ubuntu.com/7777886/ > >> # >> 8.82% postgres [kernel.kallsyms] [k] >> _raw_spin_lock_irqsave >> | >> --- _raw_spin_lock_irqsave >> | >> |--75.69%-- pagevec_lru_move_fn >> | __lru_cache_add >> | lru_cache_add >> | putback_lru_page >> | migrate_pages >> | migrate_misplaced_page >> | do_numa_page >> | handle_mm_fault >> | __do_page_fault >> | do_page_fault >> | page_fault > > So, the majority of the time is spent in numa page migration. Can you > disable numa_balancing? I'm not sure if your kernel version does that at > runtime or whether you need to reboot. > The kernel.numa_balancing sysctl might work. Otherwise you probably need > to boot with numa_balancing=0. > > It'd also be worthwhile to test this with numactl --interleave. > Trying out with numa_balancing=0 seemed to get essentially the same performance. Similarly wrapping postgres startup with --interleave. All this made me want to try with numa *really* disabled. So rebooted the box with "numa=off" appended to the kernel cmdline. Somewhat surprisingly (to me anyway), the numbers were essentially identical. The profile, however is quite different: Full report at http://paste.ubuntu.com/7806285/ 4.56% postgres [kernel.kallsyms] [k] _raw_spin_lock_irqsave | --- _raw_spin_lock_irqsave | |--41.89%-- try_to_wake_up | | | |--96.12%-- default_wake_function | | | | | |--99.96%-- pollwake | | | __wake_up_common | | | __wake_up_sync_key | | | sock_def_readable | | | | | | | |--99.94%-- unix_stream_sendmsg | | | | sock_sendmsg | | | | SYSC_sendto | | | | sys_sendto | | | | tracesys | | | | __libc_send | | | | pq_flush | | | | ReadyForQuery | | | | PostgresMain | | | | ServerLoop | | | | PostmasterMain | | | | main | | | | __libc_start_main | | | --0.06%-- [...] | | --0.04%-- [...] | | | |--2.87%-- wake_up_process | | | | | |--95.71%-- wake_up_sem_queue_do | | | SYSC_semtimedop | | | sys_semop | | | tracesys | | | __GI___semop | | | | | | | |--99.75%-- LWLockRelease | | | | | | | | | |--25.09%-- RecordTransactionCommit | | | | | CommitTransaction | | | | | CommitTransactionCommand | | | | | finish_xact_command.part.4 | | | | | PostgresMain | | | | | ServerLoop | | | | | PostmasterMain | | | | | main | | | | | __libc_start_main regards Mark -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance