Subject: Re: neither CPU nor IO bound, but throttled performance
To: pgsql-performance@lists.postgresql.org
References: <1b6d8b15-75eb-f00e-2d16-27f79d81eae7@gusw.net>
 <CABUevEyp0xGzdP=gbDspbrS0SyDesm2h8g+L5Gr0iDYy6v937A@mail.gmail.com>
From: Gunther Schadow <gunther@pragmaticdata.com>
Message-ID: <aeaeddc6-25c8-0ba9-fbbe-a0482798751e@pragmaticdata.com>
Date: Thu, 21 Feb 2019 10:59:44 -0500
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101
 Thunderbird/60.5.1
MIME-Version: 1.0
In-Reply-To: 
 <CABUevEyp0xGzdP=gbDspbrS0SyDesm2h8g+L5Gr0iDYy6v937A@mail.gmail.com>
Content-Type: multipart/alternative;
 boundary="------------1F170E208CA19CF73F317C32"
Content-Language: en-US
Precedence: bulk

This is a multi-part message in MIME format.
--------------1F170E208CA19CF73F317C32
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit

Thank you Magnus. 68% steal. Indeed. You probably hit the target. Yes.

That explains the discrepancy. I need to watch and understand that CPU 
credits issue.

regards,
-Gunther

On 2/21/2019 4:08, Magnus Hagander wrote:
> On Thu, Feb 21, 2019 at 12:34 AM Gunther <raj@gusw.net 
> <mailto:raj@gusw.net>> wrote:
>
>     Hi, I have an Amazon Linux based Postgresql 11 server here on a
>     t2.medium EC2 instance.
>
>     It is serving 24 worker processes that read jobs from a queue
>     (thanks to SELECT ... FOR UPDATE SKIP LOCKED!) and do jobs some of
>     which are reading and writing business data to the database,
>     others are only reading, and some don't hit the business data at
>     all, only the queue.
>
>     Everything flows quite nicely. Except, I don't understand why I
>     can't max out the CPU or the IO, instead, IO is almost negligible
>     yet the CPU is at 30% hardly hitting 50%.
>
>     Here I give you a view of top:
>
>     top - 23:17:09 up 45 days, 2:07, 4 users, load average: 20.32,
>     18.92, 13.80 Tasks: 338 total, 24 running, 111 sleeping, 0
>     stopped, 0 zombie %Cpu(s): 28.7 us, 2.5 sy, 0.0 ni, 0.0 id, 0.0
>     wa, 0.0 hi, 0.0 si, 68.7 st
>
>
> If I read that right, it's about 70% "steal". The description for this 
> is "the percentage of time spent in involuntary wait by the virtual 
> CPU or CPUs while the hypervisor was servicing another virtual 
> processor.".
>
> So basically, the CPU is spent dealing with other peoples VMs on the 
> same hardware. Welcome to the multi-tenant cloud.
>
> In particular, I believe T series instances get a limited number of 
> CPU "credits" per hours. My guess is you've hit this limit and are 
> thus being throttled. T series are not intended for persistent 
> workloads. Either way, this is probably a question better asked at the 
> Amazon EC2 forums rather than PostgreSQL as you'll find more people 
> who know the EC2 interactions there.
> -- 
>  Magnus Hagander
>  Me: https://www.hagander.net/ <http://www.hagander.net/>
>  Work: https://www.redpill-linpro.com/ <http://www.redpill-linpro.com/>

--------------1F170E208CA19CF73F317C32
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit

<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    <p>Thank you Magnus. 68% steal. Indeed. You probably hit the target.
      Yes. <br>
    </p>
    <p>That explains the discrepancy. I need to watch and understand
      that CPU credits issue.</p>
    <p>regards,<br>
      -Gunther<br>
    </p>
    <div class="moz-cite-prefix">On 2/21/2019 4:08, Magnus Hagander
      wrote:<br>
    </div>
    <blockquote type="cite"
cite="mid:CABUevEyp0xGzdP=gbDspbrS0SyDesm2h8g+L5Gr0iDYy6v937A@mail.gmail.com">
      <meta http-equiv="content-type" content="text/html; charset=UTF-8">
      <div dir="ltr">
        <div dir="ltr">
          <div dir="ltr">On Thu, Feb 21, 2019 at 12:34 AM Gunther &lt;<a
              href="mailto:raj@gusw.net" moz-do-not-send="true">raj@gusw.net</a>&gt;
            wrote:<br>
          </div>
          <div class="gmail_quote">
            <blockquote class="gmail_quote" style="margin:0px 0px 0px
              0.8ex;border-left:1px solid
              rgb(204,204,204);padding-left:1ex">
              <div bgcolor="#FFFFFF">
                <p>Hi, I have an Amazon Linux based Postgresql 11 server
                  here on a t2.medium EC2 instance.</p>
                <p>It is serving 24 worker processes that read jobs from
                  a queue (thanks to SELECT ... FOR UPDATE SKIP LOCKED!)
                  and do jobs some of which are reading and writing
                  business data to the database, others are only
                  reading, and some don't hit the business data at all,
                  only the queue. <br>
                </p>
                <p>Everything flows quite nicely. Except, I don't
                  understand why I can't max out the CPU or the IO,
                  instead, IO is almost negligible yet the CPU is at 30%
                  hardly hitting 50%.</p>
                <p>Here I give you a view of top:</p>
                <pre><font size="-1">top - 23:17:09 up 45 days,  2:07,  4 users,  load average: 20.32, 18.92, 13.80
Tasks: 338 total,  24 running, 111 sleeping,   0 stopped,   0 zombie
%Cpu(s): 28.7 us,  2.5 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si, 68.7 st</font></pre>
              </div>
            </blockquote>
            <div><br>
            </div>
            <div>If I read that right, it's about 70% "steal". The
              description for this is "the percentage of time spent in
              involuntary wait by the virtual CPU or CPUs while the
              hypervisor was servicing another virtual processor.".</div>
            <div><br>
            </div>
            <div>So basically, the CPU is spent dealing with other
              peoples VMs on the same hardware. Welcome to the
              multi-tenant cloud. </div>
            <div><br>
            </div>
            <div>In particular, I believe T series instances get a
              limited number of CPU "credits" per hours. My guess is
              you've hit this limit and are thus being throttled. T
              series are not intended for persistent workloads. Either
              way, this is probably a question better asked at the
              Amazon EC2 forums rather than PostgreSQL as you'll find
              more people who know the EC2 interactions there.</div>
            <div> </div>
          </div>
          -- <br>
          <div dir="ltr" class="gmail_signature">
            <div dir="ltr">
              <div> Magnus Hagander<br>
                 Me: <a href="http://www.hagander.net/" target="_blank"
                  moz-do-not-send="true">https://www.hagander.net/</a><br>
                 Work: <a href="http://www.redpill-linpro.com/"
                  target="_blank" moz-do-not-send="true">https://www.redpill-linpro.com/</a></div>
            </div>
          </div>
        </div>
      </div>
    </blockquote>
  </body>
</html>

--------------1F170E208CA19CF73F317C32--