Received: from malur.postgresql.org ([2a02:16a8:dc51::56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.89) (envelope-from ) id 1g15pX-0006jz-72 for pgsql-admin@arkaria.postgresql.org; Sat, 15 Sep 2018 08:21:27 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1g15pU-0002iQ-Q1 for pgsql-admin@arkaria.postgresql.org; Sat, 15 Sep 2018 08:21:24 +0000 Received: from makus.postgresql.org ([2001:4800:1501:1::229]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.89) (envelope-from ) id 1g15pU-0002iJ-8B for pgsql-admin@lists.postgresql.org; Sat, 15 Sep 2018 08:21:24 +0000 Received: from mail-oi0-x241.google.com ([2607:f8b0:4003:c06::241]) by makus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1g15pR-0002M7-5w for pgsql-admin@lists.postgresql.org; Sat, 15 Sep 2018 08:21:22 +0000 Received: by mail-oi0-x241.google.com with SMTP id y207-v6so14830566oie.13 for ; Sat, 15 Sep 2018 01:21:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-language; bh=qyHBt6NG/ZQQRJUnsFTni9gj6ti2JQDTO48rs09F8Y8=; b=Jh06I3f7Y1ORmKf3wRIwuKwo6E7fXIGuvd7GTSAGfmhgQ4W95iygwfNlfeGNRZn8nu 4fdpDVbgeEGMIoq2FNmnJSEBZeba+7S1mppqpy4OpIG2Uvutuol2NyYsdzwcQuq6XY+u K0ONU9/Vp0xmj6gdhePOJwC7VzcD5MPw0UdJWd7BesqQlWdXnluPtJijNu7B2tJrU+vn w/vcAiXA+2qKKNd4YGmpAua3RXKW/yhYFiyJMyrrBuATwFDNHeaNimW6IzazwkwUeAyE 3hH+imEudwOEbj7kS+CBOwfK9xo6ooo4KolaI5BMv0ktAhAxktAkg3LfWfyvi5/EpEy2 LELw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language; bh=qyHBt6NG/ZQQRJUnsFTni9gj6ti2JQDTO48rs09F8Y8=; b=jpigNUHy18oalJFPrlKxLU2/TP4xT4kjLcy+LVeqTUMlqKJDs+3ynRHohCZTuEv/Sc xUM/CSPtXYlZkcFKaSqchQBC7JsRbBVNIuJ0JjZ7LbBbHFlCp8QAjt40o3mxvh/WTL1Z pt8NPiAc/VQip5Ued1LGzk+XnUnMuLe6ssKMABIFZN3SP6BBA4nzMvIPyB1sKpLy9oJu VqBkOdpJht/z6MOZfssb/t0dAqse6j9YDUtMlLag1ADFrkzM4MXEf1kjLJQDO8UYWdII 4UK5T9+Oa64eS796KU0I8PYbl+wtMTpye7Us3lEHlSi3omdqZs3IUc/2yaYpR/q+bRFa DIeg== X-Gm-Message-State: APzg51AzASVT8qphhnkP3fkIxgrGvIoyZLDOYuRXRGSqD/tcfEBCm/3i 3Gb907t9yKJzFBgWg0HyiY4VPgBE X-Google-Smtp-Source: ANB0VdbeUJtvNf/+MQ19jd3oeqAskh356UaFuc3L/wYnsP/KZuBkGZNbDlQOUYzhCwRhVE1yOWzwcQ== X-Received: by 2002:aca:dec6:: with SMTP id v189-v6mr11110931oig.98.1536999680115; Sat, 15 Sep 2018 01:21:20 -0700 (PDT) Received: from [192.168.1.10] (ip70-171-116-89.no.no.cox.net. [70.171.116.89]) by smtp.googlemail.com with ESMTPSA id s11-v6sm1980449otk.7.2018.09.15.01.21.18 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 15 Sep 2018 01:21:19 -0700 (PDT) Subject: Re: how to debug the postgres performance issue To: pgsql-admin@lists.postgresql.org References: From: Ron Message-ID: <96eaac9a-83fe-b4bd-8e3a-8b1e93082f77@gmail.com> Date: Sat, 15 Sep 2018 03:21:18 -0500 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/alternative; boundary="------------A0CA5E3DDA91F4665950D244" Content-Language: en-US List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk This is a multi-part message in MIME format. --------------A0CA5E3DDA91F4665950D244 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit On 09/14/2018 04:34 AM, 范国腾 wrote: > > Hi, > > We are doing the performance test in two nodes system(active/standby). > Now we find that the disk IO is very high but the CPU is low. > On both nodes or just one (and if so, which one)? If you incrementally increase the workload, starting from the quiescent state, when do you see this happening? > Is there any method or tool to help us debug to find which function > makes this issue? > > We try to use the gprof but it does not help us. The result show the > LWLockAssign（the shared buffer init）is called the most time. The other > function is called no more that 33139. That seems no problem. > > Could you please  give any suggestion how to debug the high IO/low CPU > issue? > > Thanks > > Steven > > =================================== > > no time accumulated > > %   cumulative   self              self total > >  time seconds   seconds    calls  Ts/call  Ts/call name > >   0.00 0.00     0.00  9240928     0.00     0.00 LWLockAssign > > 0.00      0.00     0.00    33139     0.00     0.00 ShmemAlloc > > 0.00      0.00     0.00    33084     0.00     0.00 seg_alloc > > …. > -- Angular momentum makes the world go 'round. --------------A0CA5E3DDA91F4665950D244 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit On 09/14/2018 04:34 AM, 范国腾 wrote:

Hi,

 

We are doing the performance test in two nodes system(active/standby). Now we find that the disk IO is very high but the CPU is low.


On both nodes or just one (and if so, which one)?

If you incrementally increase the workload, starting from the quiescent state, when do you see this happening?

Is there any method or tool to help us debug to find which function makes this issue?

 

We try to use the gprof but it does not help us. The result show the LWLockAssign（the shared buffer init） is called the most time. The other function is called no more that 33139. That seems no problem.

 

Could you please  give any suggestion how to debug the high IO/low CPU issue?

 

Thanks

Steven

 

===================================

no time accumulated

 

  %   cumulative   self              self     total          

 time   seconds   seconds    calls  Ts/call  Ts/call  name   

  0.00      0.00     0.00  9240928     0.00     0.00  LWLockAssign

  0.00      0.00     0.00    33139     0.00     0.00  ShmemAlloc

  0.00      0.00     0.00    33084     0.00     0.00  seg_alloc

….


--
Angular momentum makes the world go 'round.
--------------A0CA5E3DDA91F4665950D244--