Message-ID: <46A6BBB1.1020502@denninger.net>
Date: Tue, 24 Jul 2007 21:55:45 -0500
From: Karl Denninger <karl@denninger.net>
User-Agent: Thunderbird 2.0.0.5 (Windows/20070716)
MIME-Version: 1.0
To: Tom Lane <tgl@sss.pgh.pa.us>
CC: pgsql-performance@postgresql.org
Subject: Re: Performance issue with 8.2.3 - "C" application
References: <46A6A549.1040606@denninger.net> <29633.1185331558@sss.pgh.pa.us>
In-Reply-To: <29633.1185331558@sss.pgh.pa.us>
Content-Type: multipart/alternative;
	boundary="------------090003060500060403020307"

This is a multi-part message in MIME format.
--------------090003060500060403020307
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Yeah, the problem doesn't appear to be there.  As I said, if I look at 
the PS of the system when its bogging, there aren't a whole bunch of 
processes stuck doing these, so while this does take a second or two to 
come back, that's not that bad.

Its GENERAL performance that just bites - the system is obviously out of 
CPU, but what I can't get a handle on is WHY.  It does not appear to be 
accumulating large amounts of runtime in processes I can catch, but the 
load average is quite high.

This is why I'm wondering if what I'm taking here is a hit on the 
fork/exec inside the portmaster, in the setup internally in there, in 
the IPC between my process via libPQ, etc - and how I can profile what's 
going on.

Karl Denninger (karl@denninger.net)
http://www.denninger.net


Tom Lane wrote:
> Karl Denninger <karl@denninger.net> writes:
>   
>> But.... here's the query that has a habit of taking the most time....
>>     
>
>   
>> select forum, * from post where toppost = 1 and (replied > (select 
>> lastview from forumlog where login='theuser' and forum=post.forum and 
>> number is null)) is not false AND (replied > (select lastview from 
>> forumlog where login='theuser' and forum=post.forum and 
>> number=post.number)) is not f
>> alse order by pinned desc, replied desc offset 0 limit 20
>>     
>
> Did that ever perform well for you?  It's the sub-selects that are
> likely to hurt ... in particular,
>
>   
>>          ->  Index Scan using post_top on post  (cost=0.00..57266.37 
>> rows=113 width=757)
>>                Index Cond: (toppost = 1)
>>                Filter: (((replied > (subplan)) IS NOT FALSE) AND 
>> ((replied > (subplan)) IS NOT FALSE))
>>     
>
> versus
>
>   
>>  Index Scan using post_top on post  (cost=0.00..632.03 rows=1013 width=11)
>>    Index Cond: (toppost = 1)
>>     
>
> The planner thinks that the two subplan filter conditions will eliminate
> about 90% of the rows returned by the bare indexscan (IIRC this is
> purely a rule of thumb, not based on any statistics) and that testing
> them 1013 times will add over 50000 cost units to the basic indexscan.
> That part I believe --- correlated subqueries are expensive.
>
> 			regards, tom lane
>
> ---------------------------(end of broadcast)---------------------------
> TIP 5: don't forget to increase your free space map settings
>
>
> %SPAMBLOCK-SYS: Matched [hub.org+], message ok
>   

--------------090003060500060403020307
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
  <meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Yeah, the problem doesn't appear to be there.&nbsp; As I said, if I look at
the PS of the system when its bogging, there aren't a whole bunch of
processes stuck doing these, so while this does take a second or two to
come back, that's not that bad.<br>
<br>
Its GENERAL performance that just bites - the system is obviously out
of CPU, but what I can't get a handle on is WHY.&nbsp; It does not appear to
be accumulating large amounts of runtime in processes I can catch, but
the load average is quite high.<br>
<br>
This is why I'm wondering if what I'm taking here is a hit on the
fork/exec inside the portmaster, in the setup internally in there, in
the IPC between my process via libPQ, etc - and how I can profile
what's going on.<br>
<pre class="moz-signature" cols="72">Karl Denninger (<a class="moz-txt-link-abbreviated" href="mailto:karl@denninger.net">karl@denninger.net</a>)
<a class="moz-txt-link-freetext" href="http://www.denninger.net">http://www.denninger.net</a>

</pre>
<br>
<br>
Tom Lane wrote:
<blockquote cite="mid:29633.1185331558@sss.pgh.pa.us" type="cite">
  <pre wrap="">Karl Denninger <a class="moz-txt-link-rfc2396E" href="mailto:karl@denninger.net">&lt;karl@denninger.net&gt;</a> writes:
  </pre>
  <blockquote type="cite">
    <pre wrap="">But.... here's the query that has a habit of taking the most time....
    </pre>
  </blockquote>
  <pre wrap=""><!---->
  </pre>
  <blockquote type="cite">
    <pre wrap="">select forum, * from post where toppost = 1 and (replied &gt; (select 
lastview from forumlog where login='theuser' and forum=post.forum and 
number is null)) is not false AND (replied &gt; (select lastview from 
forumlog where login='theuser' and forum=post.forum and 
number=post.number)) is not f
alse order by pinned desc, replied desc offset 0 limit 20
    </pre>
  </blockquote>
  <pre wrap=""><!---->
Did that ever perform well for you?  It's the sub-selects that are
likely to hurt ... in particular,

  </pre>
  <blockquote type="cite">
    <pre wrap="">         -&gt;  Index Scan using post_top on post  (cost=0.00..57266.37 
rows=113 width=757)
               Index Cond: (toppost = 1)
               Filter: (((replied &gt; (subplan)) IS NOT FALSE) AND 
((replied &gt; (subplan)) IS NOT FALSE))
    </pre>
  </blockquote>
  <pre wrap=""><!---->
versus

  </pre>
  <blockquote type="cite">
    <pre wrap=""> Index Scan using post_top on post  (cost=0.00..632.03 rows=1013 width=11)
   Index Cond: (toppost = 1)
    </pre>
  </blockquote>
  <pre wrap=""><!---->
The planner thinks that the two subplan filter conditions will eliminate
about 90% of the rows returned by the bare indexscan (IIRC this is
purely a rule of thumb, not based on any statistics) and that testing
them 1013 times will add over 50000 cost units to the basic indexscan.
That part I believe --- correlated subqueries are expensive.

			regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 5: don't forget to increase your free space map settings


%SPAMBLOCK-SYS: Matched [hub.org+], message ok
  </pre>
</blockquote>
</body>
</html>

--------------090003060500060403020307--


%SPAMBLOCK-SYS: Matched [@postgresql.org+], message ok