Content-Type: multipart/alternative;
 boundary="------------y90dWmQ9TPc7eVmrHO4SLsoX"
Message-ID: <d926084d-63a7-854b-2d8d-459d33ec7f70@gmail.com>
Date: Tue, 6 Dec 2022 08:51:21 -0600
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.2.2
Subject: Re: need suggestion on querying big tables
Content-Language: en-US
To: pgsql-admin@lists.postgresql.org
References: 
 <CAJuN1jj6NQDUjbMH=n4zry10JMJEZ8bRi=j6uA-3_108-Tq9cQ@mail.gmail.com>
From: Ron <ronljohnsonjr@gmail.com>
In-Reply-To: 
 <CAJuN1jj6NQDUjbMH=n4zry10JMJEZ8bRi=j6uA-3_108-Tq9cQ@mail.gmail.com>
Archived-At: 
 <https://www.postgresql.org/message-id/d926084d-63a7-854b-2d8d-459d33ec7f70%40gmail.com>
Precedence: bulk

This is a multi-part message in MIME format.
--------------y90dWmQ9TPc7eVmrHO4SLsoX
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

On 12/5/22 00:29, Ebin Jozer wrote:
> Hi Team,
> in postgresql 11 version we have two tables of size 435 GB and 347 GB.
> if we query on single table or if we are doing inner join on both the big 
> tables, it is not displacing any output, it keeps running
>
> We can see the wait event is IO and directDatafile .
>
> Server Spec : 8 cores and  64GB RAM
> PG config : 53 GB(effective_cache), 12 GB(shared buffer)
>
> can you please suggest some ideas , how we can query on big tables and 
> fasten them to get the output??

https://www.postgresql.org/docs/11/runtime-config-query.html

effective_cache_size is how much RAM *a single* query gets. "When setting 
this parameter you should consider both *PostgreSQL's shared buffers*".

https://www.postgresql.org/docs/11/runtime-config-resource.html

"a reasonable starting value for *shared_buffers is 25% of the memory* in 
your system. There are some workloads where even larger settings for 
shared_buffers are effective, but because PostgreSQL also relies on the 
operating system cache, *it is unlikely that an allocation of more than 40% 
of RAM to shared_buffers will work better* than a smaller amount."

You've set effective_cache_size to 83%, when it should be at most 20%.

-- 
Angular momentum makes the world go 'round.
--------------y90dWmQ9TPc7eVmrHO4SLsoX
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 8bit

<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body>
    On 12/5/22 00:29, Ebin Jozer wrote:<br>
    <blockquote type="cite"
cite="mid:CAJuN1jj6NQDUjbMH=n4zry10JMJEZ8bRi=j6uA-3_108-Tq9cQ@mail.gmail.com">
      <meta http-equiv="content-type" content="text/html; charset=UTF-8">
      <div dir="ltr">Hi Team,
        <div>in postgresql 11 version we have two tables of size 435 GB
          and 347 GB. </div>
        <div>if we query on single table or if we are doing inner join
          on both the big tables, it is not displacing any output, it
          keeps running </div>
        <div><br>
        </div>
        <div>We can see the wait event is IO and directDatafile .</div>
        <div><br>
        </div>
        <div>Server Spec : 8 cores and  64GB RAM</div>
        <div>PG config : 53 GB(effective_cache), 12 GB(shared buffer) </div>
        <div><br>
        </div>
        <div>can you please suggest some ideas , how we can query on big
          tables and fasten them to get the output??</div>
      </div>
    </blockquote>
    <br>
    <a class="moz-txt-link-freetext" href="https://www.postgresql.org/docs/11/runtime-config-query.html">https://www.postgresql.org/docs/11/runtime-config-query.html</a><br>
    <br>
    effective_cache_size is how much RAM <b>a single</b> query gets. 
    "When setting this parameter you should consider both <b>PostgreSQL's
      shared buffers</b>".<br>
    <br>
    <a class="moz-txt-link-freetext" href="https://www.postgresql.org/docs/11/runtime-config-resource.html">https://www.postgresql.org/docs/11/runtime-config-resource.html</a><br>
    <br>
    "a reasonable starting value for <b>shared_buffers is 25% of the
      memory</b> in your system. There are some workloads where even
    larger settings for shared_buffers are effective, but because
    PostgreSQL also relies on the operating system cache, <b>it is
      unlikely that an allocation of more than 40% of RAM to
      shared_buffers will work better</b> than a smaller amount."<br>
    <br>
    You've set effective_cache_size to 83%, when it should be at most
    20%.<br>
    <br>
    <div class="moz-signature">-- <br>
      Angular momentum makes the world go 'round.</div>
  </body>
</html>

--------------y90dWmQ9TPc7eVmrHO4SLsoX--