Content-Type: multipart/alternative;
 boundary="------------0xV6QrN3ZAjZxPVhiWkA1x52"
Message-ID: <1eb233ff-90f0-529b-f0cc-fa982f668a8f@gmail.com>
Date: Tue, 12 Jul 2022 08:51:42 -0400
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.11.0
Subject: Re: Improve "select count(*)" query - takes more than 30 mins for
 some large tables
Content-Language: en-US
To: pgsql-admin@lists.postgresql.org
References: 
 <CACxU--UJY4sD2SjW8V3W0oAPuuTzE3fAvBzTaej-BmUUNJxN_Q@mail.gmail.com>
 <1B1927BA-F7DE-40E8-8F67-BCC001CD4C6B@gmail.com>
From: Mladen Gogala <gogala.mladen@gmail.com>
In-Reply-To: <1B1927BA-F7DE-40E8-8F67-BCC001CD4C6B@gmail.com>
Archived-At: 
 <https://www.postgresql.org/message-id/1eb233ff-90f0-529b-f0cc-fa982f668a8f%40gmail.com>
Precedence: bulk

This is a multi-part message in MIME format.
--------------0xV6QrN3ZAjZxPVhiWkA1x52
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

On 7/11/22 03:23, Florents Tselai wrote:
> psql “select id from my_table" | sort -u | wc -l

That will be a lot slower than just "select count(*) from my_table".  
You are delivering data to the user program (psql) and then shipping 
them to pipe and then processing the output with "wc". Depending on the 
version, PostgreSQL has very reliable parallelism and can do counting 
rather quickly. The speed of "select count(*) from my_table" depends on 
the speed of I/O. Since the table is big, it cannot be cached in the 
file system cache, so all that you have at your disposal is the raw disk 
speed. For the smaller machines, NVME is the king. For larger rigs, you 
should consider something like Pure, XTremIO or NetApp SolidFire. People 
frequently expect database to do miracles with under par hardware.

-- 
Mladen Gogala
Database Consultant
Tel: (347) 321-1217
https://dbwhisperer.wordpress.com

--------------0xV6QrN3ZAjZxPVhiWkA1x52
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 8bit

<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body>
    <div class="moz-cite-prefix">On 7/11/22 03:23, Florents Tselai
      wrote:<br>
    </div>
    <blockquote type="cite"
      cite="mid:1B1927BA-F7DE-40E8-8F67-BCC001CD4C6B@gmail.com">
      <pre class="moz-quote-pre" wrap="">psql “select id from my_table" | sort -u | wc -l   
</pre>
    </blockquote>
    <p>That will be a lot slower than just "select count(*) from
      my_table".  You are delivering data to the user program (psql) and
      then shipping them to pipe and then processing the output with
      "wc". Depending on the version, PostgreSQL has very reliable
      parallelism and can do counting rather quickly. The speed of
      "select count(*) from my_table" depends on the speed of I/O. Since
      the table is big, it cannot be cached in the file system cache, so
      all that you have at your disposal is the raw disk speed. For the
      smaller machines, NVME is the king. For larger rigs, you should
      consider something like Pure, XTremIO or NetApp SolidFire. People
      frequently expect database to do miracles with under par hardware.<br>
    </p>
    <pre class="moz-signature" cols="72">-- 
Mladen Gogala
Database Consultant
Tel: (347) 321-1217
<a class="moz-txt-link-freetext" href="https://dbwhisperer.wordpress.com">https://dbwhisperer.wordpress.com</a>
</pre>
  </body>
</html>

--------------0xV6QrN3ZAjZxPVhiWkA1x52--