Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtp (Exim 4.80) (envelope-from ) id 1X0LqZ-0000hg-4n for pgsql-performance@arkaria.postgresql.org; Fri, 27 Jun 2014 02:25:03 +0000 Received: from localhost ([127.0.0.1] helo=postgresql.org) by malur.postgresql.org with smtp (Exim 4.80) (envelope-from ) id 1X0LqX-0002Wx-RR for pgsql-performance@arkaria.postgresql.org; Fri, 27 Jun 2014 02:25:01 +0000 Received: from makus.postgresql.org ([2001:4800:1501:1::229]) by malur.postgresql.org with esmtps (TLS1.2:DHE_RSA_AES_256_CBC_SHA256:256) (Exim 4.80) (envelope-from ) id 1X0LqV-0002Wm-DT for pgsql-performance@postgresql.org; Fri, 27 Jun 2014 02:24:59 +0000 Received: from [2404:130:0:10::40:2] (helo=mail.catalyst.net.nz) by makus.postgresql.org with esmtp (Exim 4.80) (envelope-from ) id 1X0LqP-0003pl-CM for pgsql-performance@postgresql.org; Fri, 27 Jun 2014 02:24:56 +0000 Received: from localhost (localhost [127.0.0.1]) by mail.catalyst.net.nz (Postfix) with ESMTP id 8FE2432625 for ; Fri, 27 Jun 2014 14:24:49 +1200 (NZST) X-Virus-Scanned: Debian amavisd-new at catalyst.net.nz Received: from mail.catalyst.net.nz ([127.0.0.1]) by localhost (bertrand.catalyst.net.nz [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Ol+oAR+6sRUQ for ; Fri, 27 Jun 2014 14:24:48 +1200 (NZST) Received: from [IPv6:2404:130:0:1000:8849:d1a1:4e82:439d] (unknown [IPv6:2404:130:0:1000:8849:d1a1:4e82:439d]) (Authenticated sender: mark.kirkwood) by mail.catalyst.net.nz (Postfix) with ESMTPSA id 1AFD031CCE for ; Fri, 27 Jun 2014 14:24:48 +1200 (NZST) Message-ID: <53ACD6C4.9050605@catalyst.net.nz> Date: Fri, 27 Jun 2014 14:28:20 +1200 From: Mark Kirkwood User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: pgsql-performance@postgresql.org Subject: Re: 60 core performance with 9.3 References: <53ACB190.2030206@catalyst.net.nz> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Host-Lookup-Failed: Reverse DNS lookup failed for 2404:130:0:10::40:2 (failed) X-Pg-Spam-Score: -1.1 (-) List-Archive: List-Help: List-ID: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: X-Mailing-List: pgsql-performance Precedence: bulk Sender: pgsql-performance-owner@postgresql.org On 27/06/14 14:01, Scott Marlowe wrote: > On Thu, Jun 26, 2014 at 5:49 PM, Mark Kirkwood > wrote: >> I have a nice toy to play with: Dell R920 with 60 cores and 1TB ram [1]. >> >> The context is the current machine in use by the customer is a 32 core one, >> and due to growth we are looking at something larger (hence 60 cores). >> >> Some initial tests show similar pgbench read only performance to what Robert >> found here >> http://rhaas.blogspot.co.nz/2012/04/did-i-say-32-cores-how-about-64.html >> (actually a bit quicker around 400000 tps). >> >> However doing a mixed read-write workload is getting results the same or >> only marginally quicker than the 32 core machine - particularly at higher >> number of clients (e.g 200 - 500). I have yet to break out the perf toolset, >> but I'm wondering if any folk has compared 32 and 60 (or 64) core read write >> pgbench performance? > > My guess is that the read only test is CPU / memory bandwidth limited, > but the mixed test is IO bound. > > What's your iostat / vmstat / iotop etc look like when you're doing > both read only and read/write mixed? > > That was what I would have thought too, but it does not appear to be the case, here is a typical iostat: Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nvme0n1 0.00 0.00 0.00 4448.00 0.00 41.47 19.10 0.14 0.03 0.00 0.03 0.03 14.40 nvme1n1 0.00 0.00 0.00 4448.00 0.00 41.47 19.10 0.15 0.03 0.00 0.03 0.03 15.20 nvme2n1 0.00 0.00 0.00 4549.00 0.00 42.20 19.00 0.15 0.03 0.00 0.03 0.03 15.20 nvme3n1 0.00 0.00 0.00 4548.00 0.00 42.19 19.00 0.16 0.04 0.00 0.04 0.04 16.00 dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 md0 0.00 0.00 0.00 17961.00 0.00 83.67 9.54 0.00 0.00 0.00 0.00 0.00 0.00 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 My feeling is spinlock or similar, 'perf top' shows kernel find_busiest_group kernel _raw_spin_lock as the top time users. -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance