Received: from magus.postgresql.org (magus.postgresql.org [87.238.57.229]) by mail.postgresql.org (Postfix) with ESMTP id E1DDD1C0B19A for ; Fri, 20 Jul 2012 18:52:36 -0300 (ADT) Received: from mail.anatec.com ([87.84.168.242]) by magus.postgresql.org with smtp (Exim 4.72) (envelope-from ) id 1SsL7i-0006HP-2s for pgsql-performance@postgresql.org; Fri, 20 Jul 2012 21:52:36 +0000 Received: from [192.168.10.57] ([192.168.10.57]) by mail.anatec.com over TLS secured channel with Microsoft SMTPSVC(6.0.3790.4675); Fri, 20 Jul 2012 22:52:21 +0100 Message-ID: <5009D315.4020409@anatec.com> Date: Fri, 20 Jul 2012 22:52:21 +0100 From: Ioannis Anagnostopoulos User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:14.0) Gecko/20120713 Thunderbird/14.0 MIME-Version: 1.0 To: Claudio Freire CC: pgsql-performance@postgresql.org Subject: Re: A very long running query.... References: <5007FC82.9070007@anatec.com> <5009CB5A.7070801@anatec.com> <5009CD43.8000300@anatec.com> In-Reply-To: Content-Type: multipart/alternative; boundary="------------080407070507050101050000" X-OriginalArrivalTime: 20 Jul 2012 21:52:21.0336 (UTC) FILETIME=[F0819D80:01CD66C1] X-Pg-Spam-Score: -1.9 (-) X-Archive-Number: 201207/246 X-Sequence-Number: 47427 This is a multi-part message in MIME format. --------------080407070507050101050000 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit On 20/07/2012 22:33, Claudio Freire wrote: > On Fri, Jul 20, 2012 at 6:27 PM, Ioannis Anagnostopoulos > wrote: >> On 20/07/2012 22:23, Claudio Freire wrote: >>> On Fri, Jul 20, 2012 at 6:19 PM, Ioannis Anagnostopoulos >>> wrote: >>>> " -> Nested Loop (cost=0.00..20942.93 rows=53 width=144) (actual >>>> time=62.174..17783236.718 rows=387105 loops=1)" >>>> " Join Filter: (feed_all_y2012m07.message_copies.msg_id = >>>> feed_all_y2012m07.ship_pos_messages.msg_id)" >>>> " -> Append (cost=0.00..19057.93 rows=53 width=33) (actual >>>> time=62.124..5486473.545 rows=387524 loops=1)" >>> Misestimated row counts... did you try running an analyze, or upping >>> statistic targets? >> I have run analyse every so often. I think the problem is that as I get 16K >> new rows every minutes, the "stats" are always out... Possible? > > Looking at this: > > " -> Index Scan using > idx_message_copies_wk2_date_src_pos_partial on message_copies_wk2 > message_copies (cost=0.00..19057.93 rows=52 width=32) (actual > time=62.124..5486270.845 rows=387524 loops=1)" > " Index Cond: ((date_trunc('day'::text, > msg_date_rec) = '2012-07-17 00:00:00'::timestamp without time zone) > AND (src_id = 1))" > " Filter: ((date_part('day'::text, > msg_date_rec) = 17::double precision) AND (NOT (((((pos_georef1)::text > || (pos_georef2)::text) || (pos_georef3)::text) || > (pos_georef4)::text) IS NULL)) AND (((((pos_georef1)::text || > (pos_georef2)::text) || (pos_georef3)::text) || (pos_georef4)::text) > <> ''::text))" > > It's very possible. > > I think pg 9.1 had a fix for that, but I'm not sure it will help in > your case, I'd have to know what that index looks like. Here is the index: CREATE INDEX idx_message_copies_wk2_date_src_pos_partial ON feed_all_y2012m07.message_copies_wk2 USING btree (date_trunc('day'::text, msg_date_rec), src_id, (((pos_georef1::text || pos_georef2::text) || pos_georef3::text) || pos_georef4::text)) TABLESPACE archive WHERE (((pos_georef1::text || pos_georef2::text) || pos_georef3::text) || pos_georef4::text) IS NOT NULL OR NOT (((pos_georef1::text || pos_georef2::text) || pos_georef3::text) || pos_georef4::text) = ''::text; --------------080407070507050101050000 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit
On 20/07/2012 22:33, Claudio Freire wrote:
On Fri, Jul 20, 2012 at 6:27 PM, Ioannis Anagnostopoulos
<ioannis@anatec.com> wrote:

On 20/07/2012 22:23, Claudio Freire wrote:

On Fri, Jul 20, 2012 at 6:19 PM, Ioannis Anagnostopoulos
<ioannis@anatec.com> wrote:

"        ->  Nested Loop  (cost=0.00..20942.93 rows=53 width=144) (actual
time=62.174..17783236.718 rows=387105 loops=1)"
"              Join Filter: (feed_all_y2012m07.message_copies.msg_id =
feed_all_y2012m07.ship_pos_messages.msg_id)"
"              ->  Append  (cost=0.00..19057.93 rows=53 width=33) (actual
time=62.124..5486473.545 rows=387524 loops=1)"

Misestimated row counts... did you try running an analyze, or upping
statistic targets?

I have run analyse every so often. I think the problem is that as I get 16K
new rows every minutes, the "stats" are always out... Possible?


Looking at this:

"                    ->  Index Scan using
idx_message_copies_wk2_date_src_pos_partial on message_copies_wk2
message_copies  (cost=0.00..19057.93 rows=52 width=32) (actual
time=62.124..5486270.845 rows=387524 loops=1)"
"                          Index Cond: ((date_trunc('day'::text,
msg_date_rec) = '2012-07-17 00:00:00'::timestamp without time zone)
AND (src_id = 1))"
"                          Filter: ((date_part('day'::text,
msg_date_rec) = 17::double precision) AND (NOT (((((pos_georef1)::text
|| (pos_georef2)::text) || (pos_georef3)::text) ||
(pos_georef4)::text) IS NULL)) AND (((((pos_georef1)::text ||
(pos_georef2)::text) || (pos_georef3)::text) || (pos_georef4)::text)
<> ''::text))"

It's very possible.

I think pg 9.1 had a fix for that, but I'm not sure it will help in
your case, I'd have to know what that index looks like.
Here is the index:

CREATE INDEX idx_message_copies_wk2_date_src_pos_partial
  ON feed_all_y2012m07.message_copies_wk2
  USING btree
  (date_trunc('day'::text, msg_date_rec), src_id, (((pos_georef1::text || pos_georef2::text) || pos_georef3::text) || pos_georef4::text))
TABLESPACE archive
  WHERE (((pos_georef1::text || pos_georef2::text) || pos_georef3::text) || pos_georef4::text) IS NOT NULL OR NOT (((pos_georef1::text || pos_georef2::text) || pos_georef3::text) || pos_georef4::text) = ''::text;
 --------------080407070507050101050000--