Received: from localhost (unknown [200.46.204.183]) by developer.postgresql.org (Postfix) with ESMTP id 9737A2E0052 for ; Wed, 14 May 2008 18:23:55 -0300 (ADT) Received: from developer.postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.183]) (amavisd-maia, port 10024) with ESMTP id 55411-05 for ; Wed, 14 May 2008 18:23:48 -0300 (ADT) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from n66.bullet.mail.sp1.yahoo.com (n66.bullet.mail.sp1.yahoo.com [98.136.44.50]) by developer.postgresql.org (Postfix) with SMTP id 7F35A2E0050 for ; Wed, 14 May 2008 18:23:51 -0300 (ADT) Received: from [216.252.122.218] by n66.bullet.mail.sp1.yahoo.com with NNFMP; 14 May 2008 21:23:50 -0000 Received: from [69.147.65.168] by t3.bullet.sp1.yahoo.com with NNFMP; 14 May 2008 21:23:50 -0000 Received: from [127.0.0.1] by omp503.mail.sp1.yahoo.com with NNFMP; 14 May 2008 21:23:50 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 97197.68829.bm@omp503.mail.sp1.yahoo.com Received: (qmail 86106 invoked by uid 60001); 14 May 2008 21:23:49 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Received:X-Mailer:Date:From:Reply-To:Subject:To:Cc:MIME-Version:Content-Type:Message-ID; b=YH/vsAaf1zLZCC9LrolNtVufNOkQuAPaOVAasaGSpjKnFcYpM1SP3K/kJ69YM7jdmm/3cgAx3IckQs4NvgYzhOaNakbfv5xP4QYjywilsfa3uABPwpJBxbRLRrWhle31LrGuRr0zFTjufEAJUG0Ix744FxSu6OPY4MIIKOhJFRY=; Received: from [66.39.195.151] by web44806.mail.sp1.yahoo.com via HTTP; Wed, 14 May 2008 14:23:49 PDT X-Mailer: YahooMailRC/975.38 YahooMailWebService/0.7.199 Date: Wed, 14 May 2008 14:23:49 -0700 (PDT) From: Doug Eck Reply-To: Doug Eck Subject: Re: I/O on select count(*) To: Merlin Moncure Cc: pgsql-performance@postgresql.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="0-687963745-1210800229=:85729" Message-ID: <811584.85729.qm@web44806.mail.sp1.yahoo.com> X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=0.001 tagged_above=0 required=5 tests=HTML_MESSAGE=0.001 X-Spam-Level: X-Archive-Number: 200805/185 X-Sequence-Number: 30153 --0-687963745-1210800229=:85729 Content-Type: text/plain; charset=us-ascii ----- Original Message ---- From: Merlin Moncure To: Doug Eck Cc: pgsql-performance@postgresql.org Sent: Wednesday, May 14, 2008 3:38:23 PM Subject: Re: [PERFORM] I/O on select count(*) On Wed, May 14, 2008 at 4:09 PM, Doug Eck wrote: > I have a large table (~ 2B rows) that contains an indexed timestamp column. > I am attempting to run a query to determine the number of rows for a given > day using something like "select count(*) from tbl1 where ts between > '2008-05-12 00:00:00.000' and '2008-05-12 23:59:59.999'". Explain tells me > that the query will be done using an index scan (as I would expect), and I > realize that it is going to take a while. My question concerns some unusual > I/O activity on the box (SUSE) when I run the query. > > For the first couple of minutes I see reads only. After that vmstat shows > mixed reads and writes in a ratio of about 1 block read to 5 blocks > written. We have determined that files in our data and log partitions are > being hit, but the file system itself is not growing during this time (it > appears to be writing over the same chunk of space over and over again). > Memory on the box is not being swapped while all of this is happening. I > would have guessed that a "select count(*)" would not require a bunch of > writes, and I can't begin to figure out why the number of blocks written are > so much higher than the blocks read. If I modify the where clause to only > count the rows for a given minute or two, I see the reads but I never see > the unusual write behavior. > > Any thoughts into what could be going on? Thanks in advance for your help. can you post the exact output of explain analyze? (or, at least, explain if the query takes too long) merlin The query takes a long time to run, so I'll start with the explain output. I can run explain analyze (given enough time) if you believe its output could hold some clues. db_2008=> explain select count(*) from ot_2008_05 where transact_time between '2008-05-12 00:00:00.000' and '2008-05-12 23:59:59.999'; QUERY PLAN --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Aggregate (cost=10368613.47..10368613.48 rows=1 width=0) -> Index Scan using ot_2008_05_ak2 on ot_2008_05 (cost=0.00..10011333.27 rows=142912078 width=0) Index Cond: ((transact_time >= '2008-05-12 00:00:00-04'::timestamp with time zone) AND (transact_time <= '2008-05-12 23:59:59.999-04'::timestamp with time zone)) (3 rows) db_2008=> Doug --0-687963745-1210800229=:85729 Content-Type: text/html; charset=us-ascii


----- Original Message ----
From: Merlin Moncure <mmoncure@gmail.com>
To: Doug Eck <deck1@yahoo.com>
Cc: pgsql-performance@postgresql.org
Sent: Wednesday, May 14, 2008 3:38:23 PM
Subject: Re: [PERFORM] I/O on select count(*)

On Wed, May 14, 2008 at 4:09 PM, Doug Eck <deck1@yahoo.com> wrote:
> I have a large table (~ 2B rows) that contains an indexed timestamp column.
> I am attempting to run a query to determine the number of rows for a given
> day using something like "select count(*) from tbl1 where ts between
> '2008-05-12 00:00:00.000' and '2008-05-12 23:59:59.999'".  Explain tells me
> that the query will be done using an index scan (as I would expect), and I
> realize that it is going to take a while.  My question concerns some unusual
> I/O activity on the box (SUSE)  when I run the query.
>
> For the first couple of minutes I see reads only.  After that vmstat shows
> mixed reads and writes in a ratio of about 1 block read to 5 blocks
> written.  We have determined that files in our data and log partitions are
> being hit, but the file system itself is not growing during this time (it
> appears to be writing over the same chunk of space over and over again).
> Memory on the box is not being swapped while all of this is happening.  I
> would have guessed that a "select count(*)" would not require a bunch of
> writes, and I can't begin to figure out why the number of blocks written are
> so much higher than the blocks read.  If I modify the where clause to only
> count the rows for a given minute or two, I see the reads but I never see
> the unusual write behavior.
>
> Any thoughts into what could be going on?  Thanks in advance for your help.

can you post the exact output of explain analyze? (or, at least,
explain if the query takes too long)

merlin

The query takes a long time to run, so I'll start with the explain output.  I can run explain analyze (given enough time) if you believe its output could hold some clues.

db_2008=> explain select count(*) from ot_2008_05 where transact_time between '2008-05-12 00:00:00.000' and '2008-05-12 23:59:59.999';
                                                                                QUERY PLAN
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Aggregate  (cost=10368613.47..10368613.48 rows=1 width=0)
   ->  Index Scan using ot_2008_05_ak2 on ot_2008_05  (cost=0.00..10011333.27 rows=142912078 width=0)
         Index Cond: ((transact_time >= '2008-05-12 00:00:00-04'::timestamp with time zone) AND (transact_time <= '2008-05-12 23:59:59.999-04'::timestamp with time zone))
(3 rows)

db_2008=>

Doug


--0-687963745-1210800229=:85729--