pg.ddx.io pgsql-performance@postgresql.org mailing list archive
help / color / mirror / Atom feedslow query
112+ messages / 51 participants
[nested] [flat]
* slow query
@ 2003-02-21 16:30 Oleg Lebedev <oleg.lebedev@waterford.org>
0 siblings, 2 replies; 112+ messages in thread
From: Oleg Lebedev @ 2003-02-21 16:30 UTC (permalink / raw)
To: pgsql-performance
Hi,
I am having problems with my master database now. It used to work
extremely good just two days ago, but then I started playing with
adding/dropping schemas and added another database and performance went
down.
I also have a replica database on the same server and when I run the
same query on it, it runs good.
Interestingly the planner statistics for this query are the same on the
master and replica databases.
However, query execution on the master database is about 4min. and on
the replica database is 6 sec.!
I VACUUM ANALYZED both databases and made sure they have same indexes on
the tables.
I don't know where else to look, but here are the schemas I have on the
master and replica database. The temp schemas must be the ones that I
created and then dropped.
master=# select * from pg_namespace;
nspname | nspowner | nspacl
------------+----------+--------
pg_catalog | 1 | {=U}
pg_toast | 1 | {=}
public | 1 | {=UC}
pg_temp_1 | 1 |
pg_temp_3 | 1 |
pg_temp_10 | 1 |
pg_temp_28 | 1 |
replica=> select * from pg_namespace;
nspname | nspowner | nspacl
------------+----------+--------
pg_catalog | 1 | {=U}
pg_toast | 1 | {=}
public | 1 | {=UC}
pg_temp_1 | 1 |
pg_temp_39 | 1 |
india | 105 |
Here is the query:
SELECT * FROM media m, speccharacter c
WHERE m.mediatype IN (SELECT objectid FROM mediatype WHERE
medianame='Audio')
AND m.mediachar = c.objectid
AND (m.activity='178746'
OR
(EXISTS (SELECT ism.objectid
FROM intsetmedia ism, set s
WHERE ism.set = s.objectid
AND ism.media = m.objectid AND s.activity='178746' )
)
OR
(EXISTS (SELECT dtrm.objectid
FROM dtrowmedia dtrm, dtrow dtr, dtcol dtc, datatable dt
WHERE dtrm.dtrow = dtr.objectid
AND dtrm.media = m.objectid
AND dtr.dtcol = dtc.objectid
AND dtc.datatable = dt.objectid
AND dt.activity = '178746')
)
)
ORDER BY medianame ASC, status DESC;
*************************************
This email may contain privileged or confidential material intended for the named recipient only.
If you are not the named recipient, delete this message and all attachments.
Any review, copying, printing, disclosure or other use is prohibited.
We reserve the right to monitor email sent through our network.
*************************************
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: slow query
@ 2003-02-22 02:20 Tom Lane <tgl@sss.pgh.pa.us>
parent: Oleg Lebedev <oleg.lebedev@waterford.org>
1 sibling, 0 replies; 112+ messages in thread
From: Tom Lane @ 2003-02-22 02:20 UTC (permalink / raw)
To: Oleg Lebedev <oleg.lebedev@waterford.org>; +Cc: pgsql-performance
Oleg Lebedev <oleg.lebedev@waterford.org> writes:
> Interestingly the planner statistics for this query are the same on the
> master and replica databases.
> However, query execution on the master database is about 4min. and on
> the replica database is 6 sec.!
What does EXPLAIN ANALYZE show in each case?
regards, tom lane
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: slow query
@ 2003-02-23 20:52 Josh Berkus <josh@agliodbs.com>
parent: Oleg Lebedev <oleg.lebedev@waterford.org>
1 sibling, 1 reply; 112+ messages in thread
From: Josh Berkus @ 2003-02-23 20:52 UTC (permalink / raw)
To: Oleg Lebedev <oleg.lebedev@waterford.org>; pgsql-performance
Oleg,
> I VACUUM ANALYZED both databases and made sure they have same indexes on
> the tables.
Have you VACUUM FULL the main database? And how about REINDEX?
> Here is the query:
> SELECT * FROM media m, speccharacter c
> WHERE m.mediatype IN (SELECT objectid FROM mediatype WHERE
> medianame='Audio')
The above should use an EXISTS clause, not IN, unless you are absolutely sure
that the subquery will never return more than 12 rows.
--
Josh Berkus
Aglio Database Solutions
San Francisco
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: slow query
@ 2003-02-23 21:05 Ryan Bradetich <rbradetich@uswest.net>
parent: Josh Berkus <josh@agliodbs.com>
0 siblings, 1 reply; 112+ messages in thread
From: Ryan Bradetich @ 2003-02-23 21:05 UTC (permalink / raw)
To: Josh Berkus <josh@agliodbs.com>; +Cc: Oleg Lebedev <oleg.lebedev@waterford.org>; pgsql-performance
On Sun, 2003-02-23 at 13:52, Josh Berkus wrote:
> Oleg,
>
> > I VACUUM ANALYZED both databases and made sure they have same indexes on
> > the tables.
>
> Have you VACUUM FULL the main database? And how about REINDEX?
>
> > Here is the query:
> > SELECT * FROM media m, speccharacter c
> > WHERE m.mediatype IN (SELECT objectid FROM mediatype WHERE
> > medianame='Audio')
>
> The above should use an EXISTS clause, not IN, unless you are absolutely sure
> that the subquery will never return more than 12 rows.
I am assuming you said this because EXISTS is faster for > 12 rows?
Interesting :)
thanks,
- Ryan
--
Ryan Bradetich <rbradetich@uswest.net>
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: slow query
@ 2003-02-23 21:47 Josh Berkus <josh@agliodbs.com>
parent: Ryan Bradetich <rbradetich@uswest.net>
0 siblings, 1 reply; 112+ messages in thread
From: Josh Berkus @ 2003-02-23 21:47 UTC (permalink / raw)
To: Ryan Bradetich <rbradetich@uswest.net>; +Cc: Oleg Lebedev <oleg.lebedev@waterford.org>; pgsql-performance
Ryan,
> > The above should use an EXISTS clause, not IN, unless you are absolutely
> > sure that the subquery will never return more than 12 rows.
>
> I am assuming you said this because EXISTS is faster for > 12 rows?
> Interesting :)
That's my rule of thumb, *NOT* any kind of relational-calculus-based truth.
Basically, one should only use IN for a subquery when one is *absolutely* sure
that the subquery will only return a handful of records, *and* the subquery
doesn't have to do an complex work like aggregating or custom function
evaluation.
You're safer using EXISTS for all subqueries, really.
--
Josh Berkus
Aglio Database Solutions
San Francisco
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: slow query
@ 2003-02-24 02:28 Tom Lane <tgl@sss.pgh.pa.us>
parent: Josh Berkus <josh@agliodbs.com>
0 siblings, 0 replies; 112+ messages in thread
From: Tom Lane @ 2003-02-24 02:28 UTC (permalink / raw)
To: Josh Berkus <josh@agliodbs.com>; +Cc: Ryan Bradetich <rbradetich@uswest.net>; Oleg Lebedev <oleg.lebedev@waterford.org>; pgsql-performance
Josh Berkus <josh@agliodbs.com> writes:
>> I am assuming you said this because EXISTS is faster for > 12 rows?
> That's my rule of thumb, *NOT* any kind of relational-calculus-based truth.
Keep in mind also that the tradeoffs will change quite a lot when PG 7.4
hits the streets, because the optimizer has gotten a lot smarter about
how to handle IN, but no smarter about EXISTS. Here's one rather silly
example using CVS tip:
regression=# explain analyze select * from tenk1 a where
regression-# unique1 in (select hundred from tenk1 b);
QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------
Merge Join (cost=486.32..504.11 rows=100 width=248) (actual time=453.19..468.86 rows=100 loops=1)
Merge Cond: ("outer".unique1 = "inner".hundred)
-> Index Scan using tenk1_unique1 on tenk1 a (cost=0.00..1571.87 rows=10000 width=244) (actual time=0.12..5.25 rows=101 loops=1)
-> Sort (cost=486.32..486.57 rows=100 width=4) (actual time=452.91..453.83 rows=100 loops=1)
Sort Key: b.hundred
-> HashAggregate (cost=483.00..483.00 rows=100 width=4) (actual time=447.59..449.80 rows=100 loops=1)
-> Seq Scan on tenk1 b (cost=0.00..458.00 rows=10000 width=4) (actual time=0.06..276.47 rows=10000 loops=1)
Total runtime: 472.06 msec
(8 rows)
regression=# explain analyze select * from tenk1 a where
regression-# exists (select 1 from tenk1 b where b.hundred = a.unique1);
QUERY PLAN
-------------------------------------------------------------------------------------------------------------------------------------
Seq Scan on tenk1 a (cost=0.00..35889.66 rows=5000 width=244) (actual time=3.69..1591.78 rows=100 loops=1)
Filter: (subplan)
SubPlan
-> Index Scan using tenk1_hundred on tenk1 b (cost=0.00..354.32 rows=100 width=0) (actual time=0.10..0.10 rows=0 loops=10000)
Index Cond: (hundred = $0)
Total runtime: 1593.88 msec
(6 rows)
The EXISTS case takes about the same time in 7.3, but the IN case is off
the charts (I got bored of waiting after 25 minutes...)
regards, tom lane
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: slow query
@ 2003-02-24 15:59 Oleg Lebedev <oleg.lebedev@waterford.org>
0 siblings, 1 reply; 112+ messages in thread
From: Oleg Lebedev @ 2003-02-24 15:59 UTC (permalink / raw)
To: Josh Berkus <josh@agliodbs.com>; pgsql-performance
Thanks everybody for your help.
VACUUM FULL did the job, and now the query performance is the same in
both databases. I am surprised that FULL option makes such a dramatic
change to the query performance: from 4min. to 5sec.!!! It also changed
planner stats from ~9 sec to ~8sec.
I haven't tried to REINDEX yet, though.
Regarding IN vs. EXISTS. The sub-query in the IN clause will always
return fewer records than 12. I tried using EXISTS instead of IN with
Postgres7.2.1 and it slowed down query performance. With postgres 7.3,
when I use EXISTS instead of IN the planner returns the same stats and
query performance does not improve. However, if I use
m.mediatype=(SELECT objectid FROM mediatype WHERE medianame='Audio')
the planner returns ~7 sec., which is the same as if I the query is
changed like this:
SELECT * FROM media m, speccharacter c, mediatype mt
WHERE mt.objectid=m.mediatype and mt.medianame='Audio'
...
So, using JOIN and =(SELECT ...) is better than using IN and EXISTS in
this case.
Thanks.
Oleg
-----Original Message-----
From: Josh Berkus [mailto:josh@agliodbs.com]
Sent: Sunday, February 23, 2003 1:53 PM
To: Oleg Lebedev; pgsql-performance@postgresql.org
Subject: Re: [PERFORM] slow query
Importance: Low
Oleg,
> I VACUUM ANALYZED both databases and made sure they have same indexes
> on the tables.
Have you VACUUM FULL the main database? And how about REINDEX?
> Here is the query:
> SELECT * FROM media m, speccharacter c
> WHERE m.mediatype IN (SELECT objectid FROM mediatype WHERE
> medianame='Audio')
The above should use an EXISTS clause, not IN, unless you are absolutely
sure
that the subquery will never return more than 12 rows.
--
Josh Berkus
Aglio Database Solutions
San Francisco
---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster
*************************************
This email may contain privileged or confidential material intended for the named recipient only.
If you are not the named recipient, delete this message and all attachments.
Any review, copying, printing, disclosure or other use is prohibited.
We reserve the right to monitor email sent through our network.
*************************************
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: slow query
@ 2003-02-24 16:58 Robert Treat <xzilla@users.sourceforge.net>
parent: Oleg Lebedev <oleg.lebedev@waterford.org>
0 siblings, 1 reply; 112+ messages in thread
From: Robert Treat @ 2003-02-24 16:58 UTC (permalink / raw)
To: Oleg Lebedev <oleg.lebedev@waterford.org>; +Cc: Josh Berkus <josh@agliodbs.com>; pgsql-performance
On Mon, 2003-02-24 at 10:59, Oleg Lebedev wrote:
> Thanks everybody for your help.
> VACUUM FULL did the job, and now the query performance is the same in
> both databases. I am surprised that FULL option makes such a dramatic
> change to the query performance: from 4min. to 5sec.!!! It also changed
> planner stats from ~9 sec to ~8sec.
If your seeing wildly dramatic improvments from vacuum full, you might
want to look into running regular vacuums more often (especially for
high turnover tables), increase your max_fsm_relations to 1000, and
increasing your max_fsm_pages.
Robert Treat
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: slow query
@ 2003-02-24 17:07 Oleg Lebedev <oleg.lebedev@waterford.org>
0 siblings, 0 replies; 112+ messages in thread
From: Oleg Lebedev @ 2003-02-24 17:07 UTC (permalink / raw)
To: Robert Treat <xzilla@users.sourceforge.net>; +Cc: pgsql-performance
I run VACUUM (not FULL though) every night, which I thought would be
enough for good query performance. Moreover, the data in table does not
really change significantly since database usage is pretty low right
now, therefore I thought that VACUUM FULL was an overkill.
I think that creating, populating and dropping schemas in the master
database could have affected the query performance and required VACUUM
FULL.
I will definitely look at max_fsm_relations and max_fsm_pages parameter
settings.
Thank you.
Oleg Lebedev
-----Original Message-----
From: Robert Treat [mailto:xzilla@users.sourceforge.net]
Sent: Monday, February 24, 2003 9:59 AM
To: Oleg Lebedev
Cc: Josh Berkus; pgsql-performance@postgresql.org
Subject: Re: [PERFORM] slow query
On Mon, 2003-02-24 at 10:59, Oleg Lebedev wrote:
> Thanks everybody for your help.
> VACUUM FULL did the job, and now the query performance is the same in
> both databases. I am surprised that FULL option makes such a dramatic
> change to the query performance: from 4min. to 5sec.!!! It also
> changed planner stats from ~9 sec to ~8sec.
If your seeing wildly dramatic improvments from vacuum full, you might
want to look into running regular vacuums more often (especially for
high turnover tables), increase your max_fsm_relations to 1000, and
increasing your max_fsm_pages.
Robert Treat
*************************************
This email may contain privileged or confidential material intended for the named recipient only.
If you are not the named recipient, delete this message and all attachments.
Any review, copying, printing, disclosure or other use is prohibited.
We reserve the right to monitor email sent through our network.
*************************************
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: slow query
@ 2003-02-24 17:27 Clarence Gardner <clarence@silcom.com>
parent: Robert Treat <xzilla@users.sourceforge.net>
0 siblings, 3 replies; 112+ messages in thread
From: Clarence Gardner @ 2003-02-24 17:27 UTC (permalink / raw)
To: Robert Treat <xzilla@users.sourceforge.net>; +Cc: Oleg Lebedev <oleg.lebedev@waterford.org>; Josh Berkus <josh@agliodbs.com>; pgsql-performance
On 24 Feb 2003, Robert Treat wrote:
>
> If your seeing wildly dramatic improvments from vacuum full, you might
> want to look into running regular vacuums more often (especially for
> high turnover tables), increase your max_fsm_relations to 1000, and
> increasing your max_fsm_pages.
I don't know about the settings you mention, but a frequent vacuum
does not at all obviate a vacuum full. My database is vacuumed every
night, but a while ago I found that a vacuum full changed a simple
single-table query from well over 30 seconds to one or two. We now
do a vacuum full every night.
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: slow query
@ 2003-02-24 17:44 Oleg Lebedev <oleg.lebedev@waterford.org>
0 siblings, 0 replies; 112+ messages in thread
From: Oleg Lebedev @ 2003-02-24 17:44 UTC (permalink / raw)
To: Clarence Gardner <clarence@silcom.com>; +Cc: pgsql-performance
What about REINDEXing and other DB tricks?
Do you use any of these regularly?
Thanks.
Oleg
-----Original Message-----
From: Clarence Gardner [mailto:clarence@silcom.com]
Sent: Monday, February 24, 2003 10:28 AM
To: Robert Treat
Cc: Oleg Lebedev; Josh Berkus; pgsql-performance@postgresql.org
Subject: Re: [PERFORM] slow query
On 24 Feb 2003, Robert Treat wrote:
>
> If your seeing wildly dramatic improvments from vacuum full, you might
> want to look into running regular vacuums more often (especially for
> high turnover tables), increase your max_fsm_relations to 1000, and
> increasing your max_fsm_pages.
I don't know about the settings you mention, but a frequent vacuum does
not at all obviate a vacuum full. My database is vacuumed every night,
but a while ago I found that a vacuum full changed a simple single-table
query from well over 30 seconds to one or two. We now do a vacuum full
every night.
*************************************
This email may contain privileged or confidential material intended for the named recipient only.
If you are not the named recipient, delete this message and all attachments.
Any review, copying, printing, disclosure or other use is prohibited.
We reserve the right to monitor email sent through our network.
*************************************
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: slow query
@ 2003-02-24 18:30 Robert Treat <xzilla@users.sourceforge.net>
parent: Clarence Gardner <clarence@silcom.com>
2 siblings, 1 reply; 112+ messages in thread
From: Robert Treat @ 2003-02-24 18:30 UTC (permalink / raw)
To: Clarence Gardner <clarence@silcom.com>; +Cc: Oleg Lebedev <oleg.lebedev@waterford.org>; Josh Berkus <josh@agliodbs.com>; pgsql-performance
On Mon, 2003-02-24 at 12:27, Clarence Gardner wrote:
> On 24 Feb 2003, Robert Treat wrote:
>
> >
> > If your seeing wildly dramatic improvments from vacuum full, you might
> > want to look into running regular vacuums more often (especially for
> > high turnover tables), increase your max_fsm_relations to 1000, and
> > increasing your max_fsm_pages.
>
> I don't know about the settings you mention, but a frequent vacuum
> does not at all obviate a vacuum full. My database is vacuumed every
> night, but a while ago I found that a vacuum full changed a simple
> single-table query from well over 30 seconds to one or two. We now
> do a vacuum full every night.
>
Actually if you are vacuuming frequently enough, it can (and should*)
obviate a vacuum full. Be aware that frequently enough might mean really
frequent, for instance I have several tables in my database that update
every row within a 15 minute timeframe, so I run a "lazy" vacuum on
these tables every 10 minutes. This allows postgresql to reuse the space
for these tables almost continuously so I never have to vacuum full
them.
(* this assumes your max_fsm_pages/relations settings are set correctly)
Robert Treat
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: slow query
@ 2003-02-24 18:39 Rod Taylor <rbt@rbt.ca>
parent: Clarence Gardner <clarence@silcom.com>
2 siblings, 0 replies; 112+ messages in thread
From: Rod Taylor @ 2003-02-24 18:39 UTC (permalink / raw)
To: Clarence Gardner <clarence@silcom.com>; +Cc: Robert Treat <xzilla@users.sourceforge.net>; Oleg Lebedev <oleg.lebedev@waterford.org>; Josh Berkus <josh@agliodbs.com>; pgsql-performance
On Mon, 2003-02-24 at 12:27, Clarence Gardner wrote:
> On 24 Feb 2003, Robert Treat wrote:
>
> >
> > If your seeing wildly dramatic improvments from vacuum full, you might
> > want to look into running regular vacuums more often (especially for
> > high turnover tables), increase your max_fsm_relations to 1000, and
> > increasing your max_fsm_pages.
>
> I don't know about the settings you mention, but a frequent vacuum
> does not at all obviate a vacuum full. My database is vacuumed every
> night, but a while ago I found that a vacuum full changed a simple
> single-table query from well over 30 seconds to one or two. We now
> do a vacuum full every night.
Sure is does. If your free-space-map (FSM) is up to date, tuples are
not appended to the end of the table, so table growth will not occur
(beyond a 'settling' point.
Unless you remove more data from the table than you ever expect to have
in the table again, this is enough. That said, the instant the FSM is
empty, you're table starts to grow again and a VACUUM FULL will be
required to re-shrink it.
--
Rod Taylor <rbt@rbt.ca>
PGP Key: http://www.rbt.ca/rbtpub.asc
Attachments:
[application/pgp-signature] signature.asc (186B, ../../1046111952.632.34.camel@jester/2-signature.asc)
download
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: slow query
@ 2003-02-24 18:45 Josh Berkus <josh@agliodbs.com>
parent: Robert Treat <xzilla@users.sourceforge.net>
0 siblings, 2 replies; 112+ messages in thread
From: Josh Berkus @ 2003-02-24 18:45 UTC (permalink / raw)
To: Robert Treat <xzilla@users.sourceforge.net>; Clarence Gardner <clarence@silcom.com>; +Cc: Oleg Lebedev <oleg.lebedev@waterford.org>; pgsql-performance
Robert,
> Actually if you are vacuuming frequently enough, it can (and should*)
> obviate a vacuum full. Be aware that frequently enough might mean really
> frequent, for instance I have several tables in my database that update
> every row within a 15 minute timeframe, so I run a "lazy" vacuum on
> these tables every 10 minutes. This allows postgresql to reuse the space
> for these tables almost continuously so I never have to vacuum full
> them.
This would assume absolutely perfect FSM settings, and that the DB never gets
thrown off by unexpected loads. I have never been so fortunate as to work
with such a database. However, I agree that good FSM tuning and frequent
regular VACUUMs can greatly extend the period required for running FULL.
I have not found, though, that this does anything to prevent the need for
REINDEX on frequently-updated tables. How about you, Robert?
--
Josh Berkus
Aglio Database Solutions
San Francisco
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: slow query
@ 2003-02-24 18:51 Andrew Sullivan <andrew@libertyrms.info>
parent: Clarence Gardner <clarence@silcom.com>
2 siblings, 0 replies; 112+ messages in thread
From: Andrew Sullivan @ 2003-02-24 18:51 UTC (permalink / raw)
To: pgsql-performance
On Mon, Feb 24, 2003 at 09:27:56AM -0800, Clarence Gardner wrote:
>
> I don't know about the settings you mention, but a frequent vacuum
> does not at all obviate a vacuum full. My database is vacuumed every
> night, but a while ago I found that a vacuum full changed a simple
> single-table query from well over 30 seconds to one or two. We now
> do a vacuum full every night.
This probably means that either some of your FSM settings should be
different, or that you have long-running queries, or both. _Some_
advantage to vacuum full is expected, but 30 seconds to one or two is
a pretty big deal.
Except in cases where a large percentage of the table is a vacuum
candidate, the standard vacuum should be more than adequate. But
there are a couple of gotchas.
You need to have room in your free space map to hold information
about the bulk of the to-be-freed tables. So perhaps your FSM
settings are not big enough, even though you tried to set them
higher. (Of course, if you're replacing, say, more than half the
table, setting the FSM high enough isn't practical.)
Another possibility is that you have multiple long-running
transactions that are keeping non-blocking vacuum from being very
effective. Since those transactions are possibly referencing old
versions of a row, when the non-blocking vacuum comes around, it just
skips the "dead" tuples which are nevertheless alive to someone.
(You can see the effect of this by using the contrib/pgstattuple
function). Blocking vacuum doesn't have this problem, because it
just waits on the table until everything ahead of it has committed or
rolled back. So you pay in wait time for all transactions during the
vacuum. (I have encountered this very problem on a table which gets
a lot of update activity on just on just one row. Even vacuuming
every minute, the table grew and grew, because of another misbehaving
application which was keeping a transaction open when it shouldn't
have.)
A
--
----
Andrew Sullivan 204-4141 Yonge Street
Liberty RMS Toronto, Ontario Canada
<andrew@libertyrms.info> M2P 2A8
+1 416 646 3304 x110
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: slow query
@ 2003-02-24 19:58 Robert Treat <xzilla@users.sourceforge.net>
parent: Josh Berkus <josh@agliodbs.com>
1 sibling, 0 replies; 112+ messages in thread
From: Robert Treat @ 2003-02-24 19:58 UTC (permalink / raw)
To: Josh Berkus <josh@agliodbs.com>; +Cc: Clarence Gardner <clarence@silcom.com>; Oleg Lebedev <oleg.lebedev@waterford.org>; pgsql-performance
On Mon, 2003-02-24 at 13:45, Josh Berkus wrote:
> Robert,
>
> > Actually if you are vacuuming frequently enough, it can (and should*)
> > obviate a vacuum full. Be aware that frequently enough might mean really
> > frequent, for instance I have several tables in my database that update
> > every row within a 15 minute timeframe, so I run a "lazy" vacuum on
> > these tables every 10 minutes. This allows postgresql to reuse the space
> > for these tables almost continuously so I never have to vacuum full
> > them.
>
> This would assume absolutely perfect FSM settings, and that the DB never gets
> thrown off by unexpected loads. I have never been so fortunate as to work
> with such a database. However, I agree that good FSM tuning and frequent
> regular VACUUMs can greatly extend the period required for running FULL.
>
It's somewhat relative. On one of my tables, it has about 600 rows, each
row gets updated within 15 minutes. I vacuum it every 10 minutes, which
should leave me with around 1000 tuples (dead and alive) for that table,
Even if something overload the updates on that table, chances are that I
wouldn't see enough of a performance drop to warrant a vacuum full. Of
course, its a small table, so YMMV. I think the point is though that if
your running nightly vacuum fulls just to stay ahead of the game, your
not maintaining the database optimally.
> I have not found, though, that this does anything to prevent the need for
> REINDEX on frequently-updated tables. How about you, Robert?
>
Well, this touches on a different topic. On those tables where I get
"index bloat", I do have to do REINDEX's. But that's not currently
solvable with vacuum (remember indexes dont even use FSM) though IIRC
Tom & Co. have done some work toward this for 7.4
Robert Treat
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: slow query
@ 2003-02-24 20:50 Tom Lane <tgl@sss.pgh.pa.us>
parent: Josh Berkus <josh@agliodbs.com>
1 sibling, 1 reply; 112+ messages in thread
From: Tom Lane @ 2003-02-24 20:50 UTC (permalink / raw)
To: Josh Berkus <josh@agliodbs.com>; +Cc: Robert Treat <xzilla@users.sourceforge.net>; Clarence Gardner <clarence@silcom.com>; Oleg Lebedev <oleg.lebedev@waterford.org>; pgsql-performance
Josh Berkus <josh@agliodbs.com> writes:
> ... However, I agree that good FSM tuning and frequent
> regular VACUUMs can greatly extend the period required for running FULL.
> I have not found, though, that this does anything to prevent the need for
> REINDEX on frequently-updated tables. How about you, Robert?
As of 7.3, FSM doesn't have anything to do with indexes. If you have
index bloat, it's because of the inherent inability of btree indexes to
reuse space when the data distribution changes over time. (Portions of
the btree may become empty, but they aren't recycled.) You'll
particularly get burnt by indexes that are on OIDs or sequentially
assigned ID numbers, since the set of IDs in use just naturally tends to
migrate higher over time. I don't think that the update rate per se has
much to do with this, it's the insertion of new IDs and deletion of old
ones that causes the statistical shift. The tree grows at the right
edge, but doesn't shrink at the left.
As of CVS tip, however, the situation is different ;-). Btree indexes
will recycle space using FSM in 7.4.
regards, tom lane
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: slow query
@ 2003-02-25 15:28 Robert Treat <xzilla@users.sourceforge.net>
parent: Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 112+ messages in thread
From: Robert Treat @ 2003-02-25 15:28 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Josh Berkus <josh@agliodbs.com>; Clarence Gardner <clarence@silcom.com>; Oleg Lebedev <oleg.lebedev@waterford.org>; pgsql-performance
On Mon, 2003-02-24 at 15:50, Tom Lane wrote:
> You'll
> particularly get burnt by indexes that are on OIDs or sequentially
> assigned ID numbers, since the set of IDs in use just naturally tends to
> migrate higher over time. I don't think that the update rate per se has
> much to do with this, it's the insertion of new IDs and deletion of old
> ones that causes the statistical shift.
Would it be safe to say that tables with high update rates where the
updates do not change the indexed value would not suffer from index
bloat? For example updates to non-index columns or updates that
overwrite, but don't change the value of indexed columns; do these even
need to touch the index?
Robert Treat
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: slow query
@ 2003-02-25 16:07 Tom Lane <tgl@sss.pgh.pa.us>
parent: Robert Treat <xzilla@users.sourceforge.net>
0 siblings, 1 reply; 112+ messages in thread
From: Tom Lane @ 2003-02-25 16:07 UTC (permalink / raw)
To: Robert Treat <xzilla@users.sourceforge.net>; +Cc: Josh Berkus <josh@agliodbs.com>; Clarence Gardner <clarence@silcom.com>; Oleg Lebedev <oleg.lebedev@waterford.org>; pgsql-performance
Robert Treat <xzilla@users.sourceforge.net> writes:
> Would it be safe to say that tables with high update rates where the
> updates do not change the indexed value would not suffer from index
> bloat?
I would expect not. If you vacuum often enough to keep the main table
size under control, the index should stay under control too.
> For example updates to non-index columns or updates that
> overwrite, but don't change the value of indexed columns; do these even
> need to touch the index?
Yes, they do. Think MVCC.
regards, tom lane
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: slow query
@ 2003-02-25 16:36 Peter Childs <blue.dragon@blueyonder.co.uk>
parent: Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 0 replies; 112+ messages in thread
From: Peter Childs @ 2003-02-25 16:36 UTC (permalink / raw)
To: ; +Cc: pgsql-performance
On Tue, 25 Feb 2003, Tom Lane wrote:
> Robert Treat <xzilla@users.sourceforge.net> writes:
> > Would it be safe to say that tables with high update rates where the
> > updates do not change the indexed value would not suffer from index
> > bloat?
>
> I would expect not. If you vacuum often enough to keep the main table
> size under control, the index should stay under control too.
Yes and No, From what I can work out the index is a tree and if
many updates occus the tree can become unbalanced (eventually it get so
unbalanced its no better than a seq scan.... Its not big its just all the
data is all on one side of the tree. Which is why Reindexing is a good
plan. What is really needed is a quicker way of rebalancing the tree. So
the database notices when the index is unbalanced and picks a new root
node and hangs the old root to that. (Makes for a very intresting
algorithim if I remeber my University lectures....)
Now I'm trying to sort out a very large static table that I've
just finished updating. I am beginning to think that the quickest way of
sorting it out is to dump and reload it. But I'm trying a do it in place
method. (Of Reindex it, vaccum full analyse) but what is the correct order
to do this in?
Reindex, Vaccum
or
Vaccum, Reindex.
Peter Childs
>
> > For example updates to non-index columns or updates that
> > overwrite, but don't change the value of indexed columns; do these even
> > need to touch the index?
>
> Yes, they do. Think MVCC.
>
> regards, tom lane
>
> ---------------------------(end of broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
>
> http://www.postgresql.org/users-lounge/docs/faq.html
>
^ permalink raw reply [nested|flat] 112+ messages in thread
* Slow query
@ 2003-03-24 17:48 Oleg Lebedev <oleg.lebedev@waterford.org>
0 siblings, 4 replies; 112+ messages in thread
From: Oleg Lebedev @ 2003-03-24 17:48 UTC (permalink / raw)
To: pgsql-performance
Please help me speed up the following query. It used to run in 2-5 sec.,
but now it takes 2-3 mins!
I ran VACUUM FULL ANALYZE and REINDEX.
SELECT * FROM media m
WHERE m.mediatype = (SELECT objectid FROM mediatype WHERE
medianame='Audio')
AND EXISTS
(SELECT * FROM
(SELECT objectid AS mediaid
FROM media
WHERE activity='347667'
UNION
SELECT ism.media AS mediaid
FROM intsetmedia ism, set s
WHERE ism.set = s.objectid
AND s.activity='347667' ) AS a1
WHERE a1.mediaid = m.objectid
LIMIT 1)
ORDER BY medianame ASC, status DESC
Basically it tries to find all Audios that are either explicitly
attached to the given activity, or attached to the given activity via a
many-to-many relationship intsetmedia which links records in table
Interaction, Set, and Media.
I attached the output of EXPLAIN and schemas and indexes on the tables
involved. Most of the fields are not relevant to the query, but I listed
them anyways. I discarded trigger information, though.
Thanks for your help.
Oleg
*************************************
This email may contain privileged or confidential material intended for the named recipient only.
If you are not the named recipient, delete this message and all attachments.
Any review, copying, printing, disclosure or other use is prohibited.
We reserve the right to monitor email sent through our network.
*************************************
Attachments:
[text/text] plan.txt (1.3K, ../../993DBE5B4D02194382EC8DF8554A5273113E5A@postoffice.waterford.org/3-plan.txt)
download | inline:
QUERY PLAN
Sort (cost=553657.66..553662.57 rows=1963 width=218)
Sort Key: medianame, status
InitPlan
-> Seq Scan on mediatype (cost=0.00..1.29 rows=1 width=8)
Filter: (medianame = 'Audio'::character varying)
-> Index Scan using media_mtype_index on media m (cost=0.00..553550.28 rows=1963 width=218)
Index Cond: (mediatype = $0)
Filter: (subplan)
SubPlan
-> Limit (cost=138.92..138.93 rows=1 width=24)
-> Subquery Scan a1 (cost=138.92..138.93 rows=1 width=24)
-> Unique (cost=138.92..138.93 rows=1 width=24)
-> Sort (cost=138.92..138.93 rows=2 width=24)
Sort Key: mediaid
-> Append (cost=0.00..138.91 rows=2 width=24)
-> Subquery Scan "*SELECT* 1" (cost=0.00..5.11 rows=1 width=8)
-> Index Scan using media_pkey on media (cost=0.00..5.11 rows=1 width=8)
Index Cond: (objectid = $1)
Filter: (activity = 347667::bigint)
-> Subquery Scan "*SELECT* 2" (cost=24.25..133.80 rows=1 width=24)
-> Hash Join (cost=24.25..133.80 rows=1 width=24)
Hash Cond: ("outer"."set" = "inner".objectid)
-> Index Scan using intsetmedia_media_index on intsetmedia ism (cost=0.00..109.26 rows=38 width=16)
Index Cond: (media = $1)
-> Hash (cost=24.24..24.24 rows=6 width=8)
-> Index Scan using set_act_index on "set" s (cost=0.00..24.24 rows=6 width=8)
Index Cond: (activity = 347667::bigint)
[text/text] schemas.txt (4.9K, ../../993DBE5B4D02194382EC8DF8554A5273113E5A@postoffice.waterford.org/4-schemas.txt)
download | inline:
Table "public.intsetmedia"
Column | Type | Modifiers
---------------+---------+-----------
interaction | bigint | not null
set | bigint | not null
media | bigint | not null
objectid | bigint | not null
objectversion | integer | not null
Indexes: intsetmedia_pkey primary key btree (objectid),
intsetmedia_ism_index unique btree (interaction, "set", media),
intsetmedia_media_index btree (media),
intsetmedia_set_index btree ("set")
Table "public.interaction"
Column | Type | Modifiers
----------------+------------------------+------------------------
intdesc | character varying(256) | not null default ''
state | bigint |
inttype | character varying(32) | not null default ''
number | integer | not null
video | bigint |
needsdatatable | boolean | not null default false
objectid | bigint | not null
objectversion | integer | not null
Indexes: interaction_pkey primary key btree (objectid)
Table "public.set"
Column | Type |
Modifiers
------------------+-----------------------------+-----------------------
------------------+-----------------------------+---------------------------
reference | integer | not null
activity | bigint | not null
scene | bigint |
setdesc | character varying(4000) | not null default 'No set
description available.'
acttype | bigint | not null
menutype | bigint | not null
remediate | boolean | default false
normal | boolean | default false
playpractice | boolean | default false
firsttimeonly | boolean | default false
avetime | integer |
menuimage | character varying(256) | default ''
scores | boolean | default false
minrun | integer |
maxrun | integer |
ignoretype | boolean | default false
automaticity | integer |
review | boolean | default false
learningobj | character varying(256) | default ''
setname | character varying(64) | default ''
objectid | bigint | not null
objectversion | integer | not null
icon | character varying(64) |
codestatus | bigint | not null
statuschangedate | timestamp without time zone |
considerations | character varying(256) |
Indexes: set_pkey primary key btree (objectid),
set_act_index btree (activity)
Table "public.media"
Column | Type | Modifiers
------------------+-----------------------------+---------------------
mediatype | bigint | not null
mediachar | bigint |
status | character varying(30) | not null default ''
sort | real | not null default 0
format | character varying(32) | default ''
filename | character varying(256) | not null default ''
mediadesc | character varying(4000) | default ''
medianame | character varying(4000) | not null default ''
video | bigint |
highlight | integer | default 0
highlighter | character varying(32) | default ''
activity | bigint | not null
lastupdatedby | character varying(30) | default ''
highlightedmedia | bigint |
arttype | bigint |
waitsyncmedia | bigint |
key | character varying(32) | default ''
productioncount | integer | default 0
objectid | bigint | not null
objectversion | integer | not null
cvspathstatus | bigint | not null
updatedate | timestamp without time zone |
Indexes: media_pkey primary key btree (objectid),
media_act_index btree (activity),
media_fname_index btree (filename),
media_mchar_index btree (mediachar),
media_mname_index btree (medianame),
media_mtype_index btree (mediatype)
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Slow query
@ 2003-03-24 18:54 Josh Berkus <josh@agliodbs.com>
parent: Oleg Lebedev <oleg.lebedev@waterford.org>
3 siblings, 0 replies; 112+ messages in thread
From: Josh Berkus @ 2003-03-24 18:54 UTC (permalink / raw)
To: Oleg Lebedev <oleg.lebedev@waterford.org>; pgsql-performance
Oleg,
> Please help me speed up the following query. It used to run in 2-5 sec.,
> but now it takes 2-3 mins!
> I ran VACUUM FULL ANALYZE and REINDEX.
> SELECT * FROM media m
> WHERE m.mediatype = (SELECT objectid FROM mediatype WHERE
This is a repost, isn't it?
--
Josh Berkus
Aglio Database Solutions
San Francisco
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Slow query
@ 2003-03-24 19:02 Oleg Lebedev <oleg.lebedev@waterford.org>
0 siblings, 0 replies; 112+ messages in thread
From: Oleg Lebedev @ 2003-03-24 19:02 UTC (permalink / raw)
To: Josh Berkus <josh@agliodbs.com>; pgsql-performance
No, I don't believe so.
My previous question regarding performance was solved by VACUUM FULL and
REINDEX.
The current one, I believe, is more related to query structure and
planner stats.
-----Original Message-----
From: Josh Berkus [mailto:josh@agliodbs.com]
Sent: Monday, March 24, 2003 11:55 AM
To: Oleg Lebedev; pgsql-performance@postgresql.org
Subject: Re: [PERFORM] Slow query
Oleg,
> Please help me speed up the following query. It used to run in 2-5
> sec., but now it takes 2-3 mins! I ran VACUUM FULL ANALYZE and
> REINDEX. SELECT * FROM media m
> WHERE m.mediatype = (SELECT objectid FROM mediatype WHERE
This is a repost, isn't it?
--
Josh Berkus
Aglio Database Solutions
San Francisco
*************************************
This email may contain privileged or confidential material intended for the named recipient only.
If you are not the named recipient, delete this message and all attachments.
Any review, copying, printing, disclosure or other use is prohibited.
We reserve the right to monitor email sent through our network.
*************************************
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Slow query
@ 2003-03-24 19:03 Stephan Szabo <sszabo@megazone23.bigpanda.com>
parent: Oleg Lebedev <oleg.lebedev@waterford.org>
3 siblings, 0 replies; 112+ messages in thread
From: Stephan Szabo @ 2003-03-24 19:03 UTC (permalink / raw)
To: Oleg Lebedev <oleg.lebedev@waterford.org>; +Cc: pgsql-performance
On Mon, 24 Mar 2003, Oleg Lebedev wrote:
> Please help me speed up the following query. It used to run in 2-5 sec.,
> but now it takes 2-3 mins!
EXPLAIN ANALYZE output would be useful to see where the time is actually
taking place (rather than an estimate thereof).
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Slow query
@ 2003-03-24 19:20 Oleg Lebedev <oleg.lebedev@waterford.org>
0 siblings, 0 replies; 112+ messages in thread
From: Oleg Lebedev @ 2003-03-24 19:20 UTC (permalink / raw)
To: Stephan Szabo <sszabo@megazone23.bigpanda.com>; +Cc: pgsql-performance
EXPLAIN ANALYZE plan is shown below.
I also attached it as a file.
One thing that might help is that the query produces 27 rows, which is
much less than predicted 1963.
QUERY PLAN
Sort (cost=553657.66..553662.57 rows=1963 width=218) (actual
time=133036.73..133036.75 rows=27 loops=1)
Sort Key: medianame, status
InitPlan
-> Seq Scan on mediatype (cost=0.00..1.29 rows=1 width=8) (actual
time=0.12..0.14 rows=1 loops=1)
Filter: (medianame = 'Audio'::character varying)
-> Index Scan using media_mtype_index on media m (cost=0.00..553550.28
rows=1963 width=218) (actual time=5153.36..133036.00 rows=27 loops=1)
Index Cond: (mediatype = $0)
Filter: (subplan)
SubPlan
-> Limit (cost=138.92..138.93 rows=1 width=24) (actual time=2.92..2.92
rows=0 loops=44876)
-> Subquery Scan a1 (cost=138.92..138.93 rows=1 width=24) (actual
time=2.92..2.92 rows=0 loops=44876)
-> Unique (cost=138.92..138.93 rows=1 width=24) (actual
time=2.91..2.91 rows=0 loops=44876)
-> Sort (cost=138.92..138.93 rows=2 width=24) (actual time=2.91..2.91
rows=0 loops=44876)
Sort Key: mediaid
-> Append (cost=0.00..138.91 rows=2 width=24) (actual time=2.80..2.81
rows=0 loops=44876)
-> Subquery Scan "*SELECT* 1" (cost=0.00..5.11 rows=1 width=8) (actual
time=0.06..0.06 rows=0 loops=44876)
-> Index Scan using media_pkey on media (cost=0.00..5.11 rows=1
width=8) (actual time=0.05..0.05 rows=0 loops=44876)
Index Cond: (objectid = $1)
Filter: (activity = 347667::bigint)
-> Subquery Scan "*SELECT* 2" (cost=24.25..133.80 rows=1 width=24)
(actual time=2.73..2.73 rows=0 loops=44876)
-> Hash Join (cost=24.25..133.80 rows=1 width=24) (actual
time=2.72..2.72 rows=0 loops=44876)
Hash Cond: ("outer"."set" = "inner".objectid)
-> Index Scan using intsetmedia_media_index on intsetmedia ism
(cost=0.00..109.26 rows=38 width=16) (actual time=0.04..0.04 rows=1
loops=44876)
Index Cond: (media = $1)
-> Hash (cost=24.24..24.24 rows=6 width=8) (actual time=0.14..0.14
rows=0 loops=44876)
-> Index Scan using set_act_index on "set" s (cost=0.00..24.24 rows=6
width=8) (actual time=0.11..0.13 rows=2 loops=44876)
Index Cond: (activity = 347667::bigint)
Total runtime: 133037.49 msec
-----Original Message-----
From: Stephan Szabo [mailto:sszabo@megazone23.bigpanda.com]
Sent: Monday, March 24, 2003 12:04 PM
To: Oleg Lebedev
Cc: pgsql-performance@postgresql.org
Subject: Re: [PERFORM] Slow query
On Mon, 24 Mar 2003, Oleg Lebedev wrote:
> Please help me speed up the following query. It used to run in 2-5
> sec., but now it takes 2-3 mins!
EXPLAIN ANALYZE output would be useful to see where the time is actually
taking place (rather than an estimate thereof).
*************************************
This email may contain privileged or confidential material intended for the named recipient only.
If you are not the named recipient, delete this message and all attachments.
Any review, copying, printing, disclosure or other use is prohibited.
We reserve the right to monitor email sent through our network.
*************************************
Attachments:
[text/text] plan.txt (2.0K, ../../993DBE5B4D02194382EC8DF8554A5273113E5D@postoffice.waterford.org/2-plan.txt)
download | inline:
QUERY PLAN
Sort (cost=553657.66..553662.57 rows=1963 width=218) (actual time=133036.73..133036.75 rows=27 loops=1)
Sort Key: medianame, status
InitPlan
-> Seq Scan on mediatype (cost=0.00..1.29 rows=1 width=8) (actual time=0.12..0.14 rows=1 loops=1)
Filter: (medianame = 'Audio'::character varying)
-> Index Scan using media_mtype_index on media m (cost=0.00..553550.28 rows=1963 width=218) (actual time=5153.36..133036.00 rows=27 loops=1)
Index Cond: (mediatype = $0)
Filter: (subplan)
SubPlan
-> Limit (cost=138.92..138.93 rows=1 width=24) (actual time=2.92..2.92 rows=0 loops=44876)
-> Subquery Scan a1 (cost=138.92..138.93 rows=1 width=24) (actual time=2.92..2.92 rows=0 loops=44876)
-> Unique (cost=138.92..138.93 rows=1 width=24) (actual time=2.91..2.91 rows=0 loops=44876)
-> Sort (cost=138.92..138.93 rows=2 width=24) (actual time=2.91..2.91 rows=0 loops=44876)
Sort Key: mediaid
-> Append (cost=0.00..138.91 rows=2 width=24) (actual time=2.80..2.81 rows=0 loops=44876)
-> Subquery Scan "*SELECT* 1" (cost=0.00..5.11 rows=1 width=8) (actual time=0.06..0.06 rows=0 loops=44876)
-> Index Scan using media_pkey on media (cost=0.00..5.11 rows=1 width=8) (actual time=0.05..0.05 rows=0 loops=44876)
Index Cond: (objectid = $1)
Filter: (activity = 347667::bigint)
-> Subquery Scan "*SELECT* 2" (cost=24.25..133.80 rows=1 width=24) (actual time=2.73..2.73 rows=0 loops=44876)
-> Hash Join (cost=24.25..133.80 rows=1 width=24) (actual time=2.72..2.72 rows=0 loops=44876)
Hash Cond: ("outer"."set" = "inner".objectid)
-> Index Scan using intsetmedia_media_index on intsetmedia ism (cost=0.00..109.26 rows=38 width=16) (actual time=0.04..0.04 rows=1 loops=44876)
Index Cond: (media = $1)
-> Hash (cost=24.24..24.24 rows=6 width=8) (actual time=0.14..0.14 rows=0 loops=44876)
-> Index Scan using set_act_index on "set" s (cost=0.00..24.24 rows=6 width=8) (actual time=0.11..0.13 rows=2 loops=44876)
Index Cond: (activity = 347667::bigint)
Total runtime: 133037.49 msec
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Slow query
@ 2003-03-24 19:47 Josh Berkus <josh@agliodbs.com>
parent: Oleg Lebedev <oleg.lebedev@waterford.org>
3 siblings, 0 replies; 112+ messages in thread
From: Josh Berkus @ 2003-03-24 19:47 UTC (permalink / raw)
To: Stephan Szabo <sszabo@megazone23.bigpanda.com>; pgsql-performance
Stephan,
Hmmm ... I'm a bit confused by the new EXPLAIN output. Stefan, does Oleg's
output show the time for *one* subplan execution, executed for 44,000 loops,
or does it show the total time? The former would make more sense given his
query, but I'm just not sure ....
--
-Josh Berkus
Aglio Database Solutions
San Francisco
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Slow query
@ 2003-03-24 20:28 Oleg Lebedev <oleg.lebedev@waterford.org>
0 siblings, 0 replies; 112+ messages in thread
From: Oleg Lebedev @ 2003-03-24 20:28 UTC (permalink / raw)
To: josh@agliodbs.com <josh@agliodbs.com>; Stephan Szabo <sszabo@megazone23.bigpanda.com>; pgsql-performance
I decided that it might help to list the cardinalities of the pertinent
tables:
Intsetmedia: 90,000 rows
Interaction: 26,000 rows
Set: 7,000 rows
Media: 80,000 rows
-----Original Message-----
From: Josh Berkus [mailto:josh@agliodbs.com]
Sent: Monday, March 24, 2003 12:47 PM
To: Stephan Szabo; pgsql-performance@postgresql.org
Subject: Re: [PERFORM] Slow query
Importance: Low
Stephan,
Hmmm ... I'm a bit confused by the new EXPLAIN output. Stefan, does
Oleg's
output show the time for *one* subplan execution, executed for 44,000
loops,
or does it show the total time? The former would make more sense given
his
query, but I'm just not sure ....
--
-Josh Berkus
Aglio Database Solutions
San Francisco
---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster
*************************************
This email may contain privileged or confidential material intended for the named recipient only.
If you are not the named recipient, delete this message and all attachments.
Any review, copying, printing, disclosure or other use is prohibited.
We reserve the right to monitor email sent through our network.
*************************************
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Slow query
@ 2003-03-24 20:48 Tom Lane <tgl@sss.pgh.pa.us>
parent: Oleg Lebedev <oleg.lebedev@waterford.org>
3 siblings, 0 replies; 112+ messages in thread
From: Tom Lane @ 2003-03-24 20:48 UTC (permalink / raw)
To: Oleg Lebedev <oleg.lebedev@waterford.org>; +Cc: pgsql-performance
Oleg Lebedev <oleg.lebedev@waterford.org> writes:
> SELECT * FROM media m
> WHERE m.mediatype =3D (SELECT objectid FROM mediatype WHERE
> medianame=3D'Audio')=20
> AND EXISTS=20
> (SELECT * FROM=20
> (SELECT objectid AS mediaid=20
> FROM media=20
> WHERE activity=3D'347667'=20
> UNION=20
> SELECT ism.media AS mediaid=20
> FROM intsetmedia ism, set s=20
> WHERE ism.set =3D s.objectid=20
> AND s.activity=3D'347667' ) AS a1=20
> WHERE a1.mediaid =3D m.objectid=20
> LIMIT 1)=20
> ORDER BY medianame ASC, status DESC=20
Well, one observation is that the LIMIT clause is useless and probably
counterproductive; EXISTS takes only one row from the subselect anyway.
Another is that the UNION is doing it the hard way; UNION implies doing
a duplicate-elimination step, which you don't need here. UNION ALL
would be a little quicker. But what I would do is split it into two
EXISTS:
SELECT * FROM media m
WHERE m.mediatype = (SELECT objectid FROM mediatype WHERE
medianame='Audio')
AND ( EXISTS(SELECT 1
FROM media
WHERE activity='347667'
AND objectid = m.objectid)
OR EXISTS(SELECT 1
FROM intsetmedia ism, set s
WHERE ism.set = s.objectid
AND s.activity='347667'
AND ism.media = m.objectid))
ORDER BY medianame ASC, status DESC
regards, tom lane
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Slow query
@ 2003-03-24 21:46 Oleg Lebedev <oleg.lebedev@waterford.org>
0 siblings, 1 reply; 112+ messages in thread
From: Oleg Lebedev @ 2003-03-24 21:46 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: pgsql-performance
I just ran the query you sent me and attached the output of EXPLAIN
ANALYZE as TOMs_plan.txt
It did not speed up the query significantly.
It always seemed to me that UNION is faster than OR, so I tried your
suggestion to use UNION ALL with the original query without
counter-productive LIMIT 1 in EXISTS clause. This reduced the cost of
the plan by 50%, but slowed down the query. Weird ... The plan is shown
in UNION_ALL_plan.txt
AFAIK, the only change I've done since the time when the query took 3
sec. to run was adding more indexes and increasing the size of data by
about 25%. It sounds kind of stupid, but I remember that adding indexes
sometimes slowed down my queries. I will try to drop all the indexes and
add them back again one by one.
Any other ideas?
Thanks.
Oleg
-----Original Message-----
From: Tom Lane [mailto:tgl@sss.pgh.pa.us]
Sent: Monday, March 24, 2003 1:48 PM
To: Oleg Lebedev
Cc: pgsql-performance@postgresql.org
Subject: Re: [PERFORM] Slow query
Oleg Lebedev <oleg.lebedev@waterford.org> writes:
> SELECT * FROM media m
> WHERE m.mediatype =3D (SELECT objectid FROM mediatype WHERE
> medianame=3D'Audio')=20 AND EXISTS=20
> (SELECT * FROM=20
> (SELECT objectid AS mediaid=20
> FROM media=20
> WHERE activity=3D'347667'=20
> UNION=20
> SELECT ism.media AS mediaid=20
> FROM intsetmedia ism, set s=20
> WHERE ism.set =3D s.objectid=20
> AND s.activity=3D'347667' ) AS a1=20
> WHERE a1.mediaid =3D m.objectid=20
> LIMIT 1)=20
> ORDER BY medianame ASC, status DESC=20
Well, one observation is that the LIMIT clause is useless and probably
counterproductive; EXISTS takes only one row from the subselect anyway.
Another is that the UNION is doing it the hard way; UNION implies doing
a duplicate-elimination step, which you don't need here. UNION ALL
would be a little quicker. But what I would do is split it into two
EXISTS:
SELECT * FROM media m
WHERE m.mediatype = (SELECT objectid FROM mediatype WHERE
medianame='Audio')
AND ( EXISTS(SELECT 1
FROM media
WHERE activity='347667'
AND objectid = m.objectid)
OR EXISTS(SELECT 1
FROM intsetmedia ism, set s
WHERE ism.set = s.objectid
AND s.activity='347667'
AND ism.media = m.objectid))
ORDER BY medianame ASC, status DESC
regards, tom lane
*************************************
This email may contain privileged or confidential material intended for the named recipient only.
If you are not the named recipient, delete this message and all attachments.
Any review, copying, printing, disclosure or other use is prohibited.
We reserve the right to monitor email sent through our network.
*************************************
Attachments:
[text/text] TOMs_plan.txt (1.8K, ../../993DBE5B4D02194382EC8DF8554A5273113E5F@postoffice.waterford.org/2-TOMs_plan.txt)
download | inline:
Tom's Query:
SELECT * FROM media m
WHERE m.mediatype = (SELECT objectid FROM mediatype WHERE
medianame='Audio')
AND ( EXISTS(SELECT 1
FROM media
WHERE activity='347667'
AND objectid = m.objectid)
OR EXISTS(SELECT 1
FROM intsetmedia ism, set s
WHERE ism.set = s.objectid
AND s.activity='347667'
AND ism.media = m.objectid))
ORDER BY medianame ASC, status DESC
Tom's QUERY PLAN
Sort (cost=553641.43..553648.80 rows=2945 width=218) (actual time=126087.77..126087.80 rows=27 loops=1)
Sort Key: medianame, status
InitPlan
-> Seq Scan on mediatype (cost=0.00..1.29 rows=1 width=8) (actual time=0.12..0.14 rows=1 loops=1)
Filter: (medianame = 'Audio'::character varying)
-> Index Scan using media_mtype_index on media m (cost=0.00..553471.74 rows=2945 width=218) (actual time=4870.65..126087.06 rows=27 loops=1)
Index Cond: (mediatype = $0)
Filter: ((subplan) OR (subplan))
SubPlan
-> Index Scan using media_pkey on media (cost=0.00..5.11 rows=1 width=0) (actual time=0.04..0.04 rows=0 loops=44876)
Index Cond: (objectid = $1)
Filter: (activity = 347667::bigint)
-> Hash Join (cost=24.25..133.80 rows=1 width=16) (actual time=2.71..2.71 rows=0 loops=44850)
Hash Cond: ("outer"."set" = "inner".objectid)
-> Index Scan using intsetmedia_media_index on intsetmedia ism (cost=0.00..109.26 rows=38 width=8) (actual time=0.04..0.04 rows=1 loops=44850)
Index Cond: (media = $1)
-> Hash (cost=24.24..24.24 rows=6 width=8) (actual time=0.14..0.14 rows=0 loops=44850)
-> Index Scan using set_act_index on "set" s (cost=0.00..24.24 rows=6 width=8) (actual time=0.11..0.13 rows=2 loops=44850)
Index Cond: (activity = 347667::bigint)
Total runtime: 126088.33 msec
[text/text] UNION_ALL_plan.txt (2.1K, ../../993DBE5B4D02194382EC8DF8554A5273113E5F@postoffice.waterford.org/3-UNION_ALL_plan.txt)
download | inline:
New Query:
SELECT * FROM media m
WHERE m.mediatype = (SELECT objectid FROM mediatype WHERE medianame='Audio')
AND EXISTS
(SELECT * FROM
(SELECT objectid AS mediaid
FROM media
WHERE activity='347667'
UNION ALL
SELECT ism.media AS mediaid
FROM intsetmedia ism, set s
WHERE ism.set = s.objectid
AND s.activity='347667' ) AS a1
WHERE a1.mediaid = m.objectid
)
ORDER BY medianame ASC, status DESC
QUERY PLAN:
Sort (cost=280852.86..280857.77 rows=1963 width=218) (actual time=129360.72..129360.73 rows=27 loops=1)
Sort Key: medianame, status
InitPlan
-> Seq Scan on mediatype (cost=0.00..1.29 rows=1 width=8) (actual time=0.13..0.15 rows=1 loops=1)
Filter: (medianame = 'Audio'::character varying)
-> Index Scan using media_mtype_index on media m (cost=0.00..280745.48 rows=1963 width=218) (actual time=5523.85..129359.96 rows=27 loops=1)
Index Cond: (mediatype = $0)
Filter: (subplan)
SubPlan
-> Subquery Scan a1 (cost=0.00..138.91 rows=2 width=24) (actual time=2.84..2.84 rows=0 loops=44876)
-> Append (cost=0.00..138.91 rows=2 width=24) (actual time=2.84..2.84 rows=0 loops=44876)
-> Subquery Scan "*SELECT* 1" (cost=0.00..5.11 rows=1 width=8) (actual time=0.06..0.06 rows=0 loops=44876)
-> Index Scan using media_pkey on media (cost=0.00..5.11 rows=1 width=8) (actual time=0.04..0.04 rows=0 loops=44876)
Index Cond: (objectid = $1)
Filter: (activity = 347667::bigint)
-> Subquery Scan "*SELECT* 2" (cost=24.25..133.80 rows=1 width=24) (actual time=2.77..2.77 rows=0 loops=44850)
-> Hash Join (cost=24.25..133.80 rows=1 width=24) (actual time=2.76..2.76 rows=0 loops=44850)
Hash Cond: ("outer"."set" = "inner".objectid)
-> Index Scan using intsetmedia_media_index on intsetmedia ism (cost=0.00..109.26 rows=38 width=16) (actual time=0.04..0.04 rows=1 loops=44850)
Index Cond: (media = $1)
-> Hash (cost=24.24..24.24 rows=6 width=8) (actual time=0.14..0.14 rows=0 loops=44850)
-> Index Scan using set_act_index on "set" s (cost=0.00..24.24 rows=6 width=8) (actual time=0.11..0.12 rows=2 loops=44850)
Index Cond: (activity = 347667::bigint)
Total runtime: 129361.35 msec
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Slow query
@ 2003-03-24 22:08 Tom Lane <tgl@sss.pgh.pa.us>
parent: Oleg Lebedev <oleg.lebedev@waterford.org>
0 siblings, 0 replies; 112+ messages in thread
From: Tom Lane @ 2003-03-24 22:08 UTC (permalink / raw)
To: Oleg Lebedev <oleg.lebedev@waterford.org>; +Cc: pgsql-performance
Oleg Lebedev <oleg.lebedev@waterford.org> writes:
> I just ran the query you sent me and attached the output of EXPLAIN
> ANALYZE as TOMs_plan.txt
> It did not speed up the query significantly.
Nope. I was hoping to see a faster-start plan, but given the number of
rows involved I guess it won't change its mind. You're going to have to
think about a more intelligent approach, rather than minor tweaks.
One question: since objectid is evidently a primary key, why are you
doing a subselect for the first part? Wouldn't it give the same result
just to say "m.activity = '347667'" in the top-level WHERE?
As for the second part, I think you'll have to try to rewrite it as a
join with the media table.
regards, tom lane
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Slow query
@ 2003-03-24 22:37 Oleg Lebedev <oleg.lebedev@waterford.org>
0 siblings, 0 replies; 112+ messages in thread
From: Oleg Lebedev @ 2003-03-24 22:37 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: pgsql-performance
You are right. I rewrote the query using JOINs and it increased
performance from 123 sec. to 20msec. I betcha I screwed smth up, but I
list the rewritten query below anyways. I also attached the new plan.
Thank you.
SELECT * FROM media m
JOIN
((SELECT objectid AS mediaid
FROM media
WHERE activity='347667')
UNION
(SELECT ism.media AS mediaid
FROM intsetmedia ism, set s
WHERE ism.set = s.objectid
AND s.activity='347667' )) a1
ON
m.mediatype = (SELECT objectid FROM mediatype WHERE medianame='Audio')
AND m.objectid=mediaid
ORDER BY medianame ASC, status DESC
-----Original Message-----
From: Tom Lane [mailto:tgl@sss.pgh.pa.us]
Sent: Monday, March 24, 2003 3:09 PM
To: Oleg Lebedev
Cc: pgsql-performance@postgresql.org
Subject: Re: [PERFORM] Slow query
Oleg Lebedev <oleg.lebedev@waterford.org> writes:
> I just ran the query you sent me and attached the output of EXPLAIN
> ANALYZE as TOMs_plan.txt It did not speed up the query significantly.
Nope. I was hoping to see a faster-start plan, but given the number of
rows involved I guess it won't change its mind. You're going to have to
think about a more intelligent approach, rather than minor tweaks.
One question: since objectid is evidently a primary key, why are you
doing a subselect for the first part? Wouldn't it give the same result
just to say "m.activity = '347667'" in the top-level WHERE?
As for the second part, I think you'll have to try to rewrite it as a
join with the media table.
regards, tom lane
*************************************
This email may contain privileged or confidential material intended for the named recipient only.
If you are not the named recipient, delete this message and all attachments.
Any review, copying, printing, disclosure or other use is prohibited.
We reserve the right to monitor email sent through our network.
*************************************
Attachments:
[text/text] NEW_plan.txt (2.3K, ../../993DBE5B4D02194382EC8DF8554A5273113E61@postoffice.waterford.org/2-NEW_plan.txt)
download | inline:
New Query:
SELECT * FROM media m
JOIN
((SELECT objectid AS mediaid
FROM media
WHERE activity='347667')
UNION
(SELECT ism.media AS mediaid
FROM intsetmedia ism, set s
WHERE ism.set = s.objectid
AND s.activity='347667' )) a1
ON
m.mediatype = (SELECT objectid FROM mediatype WHERE medianame='Audio')
AND m.objectid=mediaid
ORDER BY medianame ASC, status DESC
QUERY PLAN
Sort (cost=1831.47..1831.47 rows=1 width=225) (actual time=19.56..19.58 rows=27 loops=1)
Sort Key: m.medianame, m.status
InitPlan
-> Seq Scan on mediatype (cost=0.00..1.29 rows=1 width=8) (actual time=0.10..0.12 rows=1 loops=1)
Filter: (medianame = 'Audio'::character varying)
-> Nested Loop (cost=1702.97..1831.46 rows=1 width=225) (actual time=16.44..18.93 rows=27 loops=1)
-> Subquery Scan a1 (cost=1702.97..1704.19 rows=24 width=24) (actual time=16.10..16.72 rows=58 loops=1)
-> Unique (cost=1702.97..1704.19 rows=24 width=24) (actual time=16.08..16.56 rows=58 loops=1)
-> Sort (cost=1702.97..1703.58 rows=243 width=24) (actual time=16.08..16.25 rows=264 loops=1)
Sort Key: mediaid
-> Append (cost=0.00..1693.32 rows=243 width=24) (actual time=0.07..9.11 rows=264 loops=1)
-> Subquery Scan "*SELECT* 1" (cost=0.00..505.76 rows=169 width=8) (actual time=0.06..0.65 rows=55 loops=1)
-> Index Scan using media_act_index on media (cost=0.00..505.76 rows=169 width=8) (actual time=0.06..0.53 rows=55 loops=1)
Index Cond: (activity = 347667::bigint)
-> Subquery Scan "*SELECT* 2" (cost=0.00..1187.57 rows=75 width=24) (actual time=0.11..8.11 rows=209 loops=1)
-> Nested Loop (cost=0.00..1187.57 rows=75 width=24) (actual time=0.10..7.68 rows=209 loops=1)
-> Index Scan using set_act_index on "set" s (cost=0.00..24.23 rows=6 width=8) (actual time=0.05..0.06 rows=2 loops=1)
Index Cond: (activity = 347667::bigint)
-> Index Scan using intsetmedia_set_index on intsetmedia ism (cost=0.00..203.42 rows=68 width=16) (actual time=0.04..3.56 rows=104 loops=2)
Index Cond: (ism."set" = "outer".objectid)
-> Index Scan using media_pkey on media m (cost=0.00..5.22 rows=1 width=217) (actual time=0.03..0.03 rows=0 loops=58)
Index Cond: (m.objectid = "outer".mediaid)
Filter: (mediatype = $0)
Total runtime: 20.36 msec
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Slow query
@ 2003-03-25 03:26 Nikolaus Dilger <nikolaus@dilger.cc>
0 siblings, 0 replies; 112+ messages in thread
From: Nikolaus Dilger @ 2003-03-25 03:26 UTC (permalink / raw)
To: oleg.lebedev@waterford.org; +Cc: pgsql-performance
Oleg,
My guess is that the query runs slow because by adding
data you exceeded what your database can do in memory
and you need to do some kind of disk sort.
How about rewriting your query without the UNION and
the EXISTS to something like
SELECT * FROM media m
WHERE m.mediatype = (SELECT objectid FROM mediatype
WHERE medianame='Audio')
AND ( m.activity='347667'
OR m.objectid IN (
SELECT s.objectid
FROM intsetmedia ism, set s
WHERE ism.set = s.objectid
AND s.activity='347667'))
ORDER BY medianame ASC, status DESC
Regards,
Nikolaus Dilger
On Mon, 24 Mar 2003, Oleg Lebedev wrote:
Message
Please help me speed
up the following query. It used to run in 2-5 sec., but
now it takes 2-3
mins!
I ran VACUUM FULL
ANALYZE and REINDEX.
SELECT * FROM media
m
WHERE m.mediatype =
(SELECT objectid FROM mediatype WHERE
medianame='Audio')
AND EXISTS
(SELECT * FROM
(SELECT objectid AS mediaid
FROM media
WHERE activity='347667'
UNION
SELECT ism.media AS mediaid
FROM intsetmedia ism, set s
WHERE ism.set = s.objectid
AND s.activity='347667' ) AS a1
WHERE a1.mediaid = m.objectid
LIMIT 1)
ORDER BY medianame ASC, status DESC
Basically it tries
to find all Audios that are either explicitly attached
to the given activity, or
attached to the given activity via a many-to-many
relationship intsetmedia which
links records in table Interaction, Set, and Media.
I attached the
output of EXPLAIN and schemas and indexes on the tables
involved. Most of the
fields are not relevant to the query, but I listed them
anyways. I discarded
trigger information, though.
Thanks for your
help.
Oleg
*************************************
This email may contain privileged or confidential
material intended for the named recipient only.
If you are not the named recipient, delete this message
and all attachments.
Any review, copying, printing, disclosure or other use
is prohibited.
We reserve the right to monitor email sent through our
network.
*************************************
^ permalink raw reply [nested|flat] 112+ messages in thread
* Slow query?
@ 2003-09-09 06:27 Waruna Geekiyanage <waruna@nirmani.com>
0 siblings, 0 replies; 112+ messages in thread
From: Waruna Geekiyanage @ 2003-09-09 06:27 UTC (permalink / raw)
To: pgsql-performance
Hi All,
Is it usual that the following query to take 22 secs with the machine I have?
Any other reason?
Hope I have provided all the details need.
Thanks,
Waruna
Tables:
/* --------------------------------------------------------
Table structure for table "tvDiary"
-------------------------------------------------------- */
CREATE TABLE "tvDiary" (
"member" int4 NOT NULL,
"timeSlot" int2 NOT NULL references "timeSlot"("code"),
"channel" varchar(4) NOT NULL references "tvChannel"("code"),
"date" date NOT NULL,
CONSTRAINT "tvDiary_pkey" PRIMARY KEY ("date", "member", "timeSlot")
);
Indexed on "date"
/* --------------------------------------------------------
Table structure for table "mDiary"
-------------------------------------------------------- */
CREATE TABLE "mDiary" (
"member" int4 NOT NULL,
"area" char(1) NOT NULL,
"district" int2 references "district"("code"),
"date" date NOT NULL,
CONSTRAINT "mDiary_pkey" PRIMARY KEY ("date", "member")
);
Indexed on "date"
# Records
tvDiary : 7 300 000
mDiary : 850 000
machine :
Celeron 1.0GHz RAM - 390MB , 40 GB IDE HDD
RedHat Linux 9
kernel.shmmni = 4096
kernel.shmall = 33554432
kernel.shmmax = 134217728
postgres 7.3.4
shared_buffers = 8192
sort_mem = 65536
Query:
SELECT COUNT(td.member) AS count, td.date AS date, td."timeSlot" AS "timeSlot", td.channel AS channel,
tg.district AS district,tg.area AS area
FROM "tvDiary" td ,(SELECT DISTINCT(md.member) AS member, md.area AS area, md.district as district
FROM "mDiary" md
WHERE (md.date BETWEEN '20020301' AND '20020330') ) AS tg
WHERE(td.date BETWEEN '20020301' AND '20020330') AND (td.member=tg.member)
GROUP BY td.date,td."timeSlot", td.channel,tg.district,tg.area;
QUERY PLAN
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
-----------------------
Aggregate (cost=91790.44..100942.65 rows=52298 width=28) (actual time=18396.42..21764.44 rows=57478 loops=1)
-> Group (cost=91790.44..99635.19 rows=522983 width=28) (actual time=18396.34..21158.23 rows=281733 loops=1)
-> Sort (cost=91790.44..93097.90 rows=522983 width=28) (actual time=18396.30..18588.91 rows=281733 loops=1)
Sort Key: td.date, td."timeSlot", td.channel, tg.district, tg.area
-> Merge Join (cost=34290.10..42116.42 rows=522983 width=28) (actual time=8159.30..10513.62 rows=281733 ops=1)
Merge Cond: ("outer".member = "inner".member)
-> Sort (cost=29121.48..29755.35 rows=253551 width=17) (actual time=6752.36..6933.38 rows=282552 loops=1)
Sort Key: td.member
-> Index Scan using d_tvdiary_key on "tvDiary" td (cost=0.00..6362.82 rows=253551 width=17) (actual time=95.80..4766.25 rows=282587
loops=1)
Index Cond: ((date >= '2002-03-01'::date) AND (date <= '2002-03-30'::date))
-> Sort (cost=5168.63..5179.26 rows=4251 width=11) (actual time=1406.88..1590.72 rows=281955 loops=1)
Sort Key: tg.member
-> Subquery Scan tg (cost=4487.31..4912.42 rows=4251 width=11) (actual time=1228.55..1397.20 rows=2348 loops=1)
-> Unique (cost=4487.31..4912.42 rows=4251 width=11) (actual time=1228.52..1390.12 rows=2348 loops=1)
-> Sort (cost=4487.31..4593.59 rows=42511 width=11) (actual time=1228.51..1257.87 rows=46206 loops=1)
Sort Key: member, area, district
-> Index Scan using d_mdiary_key on "mDiary" md (cost=0.00..1219.17 rows=42511 width=11) (actual time=60.20..750.
67 rows=46206 loops=1)
Index Cond: ((date >= '2002-03-01'::date) AND (date <= '2002-03-30'::date))
Total runtime: 21992.24 msec
(19 rows)
^ permalink raw reply [nested|flat] 112+ messages in thread
* Slow query
@ 2004-02-24 13:17 Ivan Voras <ivoras@fer.hr>
0 siblings, 1 reply; 112+ messages in thread
From: Ivan Voras @ 2004-02-24 13:17 UTC (permalink / raw)
To: pgsql-performance
I have a query that I think should run faster. The machine is P2/400
with enough ram (384MB), but still, maybe the query could be tuned up.
postgresql.conf is stock with these values changed:
fsync=false
shared_buffers = 5000
sort_mem = 8192
vacuum_mem = 16384
This is a development machine, the production will be dual P3, 1GHz, 1GB
RAM, but I fear that the execution will still be slow, as the tables
will get bigger.
I've pasted information about the database, and the explain output, but
the text is horribly wrapped so there's a clean copy on the web in
http://geri.cc.fer.hr/~ivoras/query.txt
The intention is: there is a table called cl_log which records events
from various sources, some of which also have data in data_kat_id and
data_user_id fields, some of which don't (hence the outer joins). The
query is report-style, and tries to collect as much data as possible
about the events. Tables cl_source, cl_handler and cl_event_type hold
information about the type of event. They are small (currently 1-3
records in each, will grow to about 10 records).
ferweb=> explain analyze SELECT cl_log.*, cl_source.name AS source_name,
cl_source.description AS source_description,
cl_handler.name AS handler_name, cl_handler.description AS
handler_description, cl_event_type.name AS event_type_name,
cl_event_type.description as event_type_description, users.jime,
kategorija.knaziv
FROM cl_log
INNER JOIN cl_source ON source_id=cl_source.id
INNER JOIN cl_handler ON cl_source.handler_id=cl_handler.id
INNER JOIN cl_event_type ON event_type_id=cl_event_type.id
LEFT OUTER JOIN kategorija ON data_kat_id=kategorija.id
LEFT OUTER JOIN users ON data_user_id=users.id
ORDER BY time desc LIMIT 30;
QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=788.78..788.79 rows=2 width=500) (actual
time=23229.78..23230.44 rows=30 loops=1)
-> Sort (cost=788.78..788.79 rows=3 width=500) (actual
time=23229.75..23230.10 rows=31 loops=1)
Sort Key: cl_log."time"
-> Nested Loop (cost=1.04..788.76 rows=3 width=500) (actual
time=4078.85..20185.89 rows=38999 loops=1)
-> Nested Loop (cost=1.04..771.27 rows=3 width=485)
(actual time=4078.71..14673.27 rows=38999 loops=1)
-> Hash Join (cost=1.04..754.21 rows=3
width=417) (actual time=4078.54..8974.08 rows=38999 loops=1)
Hash Cond: ("outer".event_type_id = "inner".id)
-> Nested Loop (cost=0.00..752.16 rows=195
width=288) (actual time=4078.20..6702.17 rows=38999 loops=1)
Join Filter: ("inner".handler_id =
"outer".id)
-> Seq Scan on cl_handler
(cost=0.00..1.01 rows=1 width=104) (actual time=0.02..0.04 rows=1 loops=1)
-> Materialize (cost=748.72..748.72
rows=195 width=184) (actual time=4078.08..4751.52 rows=38999 loops=1)
-> Nested Loop
(cost=0.00..748.72 rows=195 width=184) (actual time=0.21..3197.16
rows=38999 loops=1)
-> Seq Scan on cl_source
(cost=0.00..1.01 rows=1 width=108) (actual time=0.05..0.06 rows=1 loops=1)
-> Index Scan using
cl_log_source on cl_log (cost=0.00..745.27 rows=195 width=76) (actual
time=0.11..1467.08 rows=38999 loops=1)
Index Cond:
(cl_log.source_id = "outer".id)
-> Hash (cost=1.03..1.03 rows=3 width=129)
(actual time=0.12..0.12 rows=0 loops=1)
-> Seq Scan on cl_event_type
(cost=0.00..1.03 rows=3 width=129) (actual time=0.04..0.08 rows=3 loops=1)
-> Index Scan using kategorija_pkey on kategorija
(cost=0.00..5.82 rows=1 width=68) (actual time=0.05..0.07 rows=1
loops=38999)
Index Cond: ("outer".data_kat_id =
kategorija.id)
-> Index Scan using users_pkey on users
(cost=0.00..5.97 rows=1 width=15) (actual time=0.05..0.07 rows=1
loops=38999)
Index Cond: ("outer".data_user_id = users.id)
Total runtime: 23267.25 msec
(22 rows)
ferweb=> select count(*) from cl_log;
count
-------
38999
(1 row)
ferweb=> select count(*) from cl_handler;
count
-------
1
(1 row)
ferweb=> select count(*) from cl_source;
count
-------
1
(1 row)
ferweb=> select count(*) from cl_event_type;
count
-------
3
(1 row)
ferweb=> select count(*) from users;
count
-------
2636
(1 row)
ferweb=> select count(*) from kategorija;
count
-------
1928
(1 row)
--
Every sufficiently advanced magic is indistinguishable from technology
- Arthur C Anticlarke
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Slow query
@ 2004-03-01 16:26 scott.marlowe <scott.marlowe@ihs.com>
parent: Ivan Voras <ivoras@fer.hr>
0 siblings, 0 replies; 112+ messages in thread
From: scott.marlowe @ 2004-03-01 16:26 UTC (permalink / raw)
To: Ivan Voras <ivoras@fer.hr>; +Cc: pgsql-performance
On Tue, 24 Feb 2004, Ivan Voras wrote:
> -> Nested Loop (cost=1.04..788.76 rows=3 width=500) (actual
> time=4078.85..20185.89 rows=38999 loops=1)
> -> Nested Loop (cost=1.04..771.27 rows=3 width=485)
> (actual time=4078.71..14673.27 rows=38999 loops=1)
> -> Nested Loop (cost=0.00..752.16 rows=195
> width=288) (actual time=4078.20..6702.17 rows=38999 loops=1)
> -> Nested Loop
> (cost=0.00..748.72 rows=195 width=184) (actual time=0.21..3197.16
> rows=38999 loops=1)
Note those nested loops up there. They think that you are going to be
operating on 3,3,195, and 195 rows respectively, when they actually are
operating on 38999, 38999, 38999, and 38999 in reality.
set enable_nestloop = off
and see if that helps. If so, see if altering the responsible columns
default stats to something higher (100 is a good start) and reanalyze to
see if you get a better plan. As long as those estimates are that far
off, you're gonna get a poorly performing query when the planner is
allowed to use nested loops.
^ permalink raw reply [nested|flat] 112+ messages in thread
* Slow query
@ 2004-10-22 19:06 Roger Ging <rging@paccomsys.com>
0 siblings, 1 reply; 112+ messages in thread
From: Roger Ging @ 2004-10-22 19:06 UTC (permalink / raw)
To: pgsql-performance
The following query has never finished. I have let it run for over 24
hours. This is a one time update that is part of a conversion script
from MSSQL data. All of the tables are freshly built and inserted
into. I have not run explain analyze because it does not return in a
reasonable time. Explain output is posted below. Any suggestions on
syntax changes or anything else to improve this would be appreciated.
Dual PIII 1Ghz
4 GB RAM
4 spindle IDE RAID 0 on LSI controller.
Postgres 7.4.5
Linux version 2.6.3-7mdk-p3-smp-64GB
postgresql.cong snip:
tcpip_socket = true
max_connections = 40
shared_buffers = 1000
sort_mem = 65536
fsync = true
source_song_title +-10,500,000 rows
source_song +-9,500,000 rows
source_system 10 rows
source_title +- 5,600,000
Code run right before this query:
create index ssa_source_song_id on source_song_artist (source_song_id);
analyze source_song_artist;
create index sa_artist_id on source_artist (artist_id);
analyze source_artist;
create index ss_source_song_id on source_song (source_song_id);
analyze source_song;
create index st_title_id on source_title (title_id);
analyze source_title;
source_song.source_song_id = int4
source_song_title.source_song_id = int4
source_title.title_id = int4
source_song_title.title_id = int4
update source_song_title set
source_song_title_id = nextval('source_song_title_seq')
,licensing_match_order = (select licensing_match_order from
source_system where source_system_id = ss.source_system_id)
,affiliation_match_order = (select affiliation_match_order from
source_system where source_system_id = ss.source_system_id)
,title = st.title
from source_song_title sst
join source_song ss on ss.source_song_id = sst.source_song_id
join source_title st on st.title_id = sst.title_id
where source_song_title.source_song_id = sst.source_song_id;
Explain output:
"Hash Join (cost=168589.60..16651072.43 rows=6386404 width=335)"
" Hash Cond: ("outer".title_id = "inner".title_id)"
" -> Merge Join (cost=0.00..1168310.61 rows=6386403 width=311)"
" Merge Cond: ("outer".source_song_id = "inner".source_song_id)"
" -> Merge Join (cost=0.00..679279.40 rows=6386403 width=16)"
" Merge Cond: ("outer".source_song_id =
"inner".source_song_id)"
" -> Index Scan using source_song_title_pkey on
source_song_title sst (cost=0.00..381779.37 rows=10968719 width=8)"
" -> Index Scan using ss_source_song_id on source_song ss
(cost=0.00..190583.36 rows=6386403 width=8)"
" -> Index Scan using source_song_title_pkey on
source_song_title (cost=0.00..381779.37 rows=10968719 width=303)"
" -> Hash (cost=117112.08..117112.08 rows=5513808 width=32)"
" -> Seq Scan on source_title st (cost=0.00..117112.08
rows=5513808 width=32)"
" SubPlan"
" -> Seq Scan on source_system (cost=0.00..1.14 rows=2 width=4)"
" Filter: (source_system_id = $0)"
" -> Seq Scan on source_system (cost=0.00..1.14 rows=2 width=2)"
" Filter: (source_system_id = $0)"
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Slow query
@ 2004-10-22 20:37 Tom Lane <tgl@sss.pgh.pa.us>
parent: Roger Ging <rging@paccomsys.com>
0 siblings, 1 reply; 112+ messages in thread
From: Tom Lane @ 2004-10-22 20:37 UTC (permalink / raw)
To: Roger Ging <rging@paccomsys.com>; +Cc: pgsql-performance
Roger Ging <rging@paccomsys.com> writes:
> update source_song_title set
> source_song_title_id = nextval('source_song_title_seq')
> ,licensing_match_order = (select licensing_match_order from
> source_system where source_system_id = ss.source_system_id)
> ,affiliation_match_order = (select affiliation_match_order from
> source_system where source_system_id = ss.source_system_id)
> ,title = st.title
> from source_song_title sst
> join source_song ss on ss.source_song_id = sst.source_song_id
> join source_title st on st.title_id = sst.title_id
> where source_song_title.source_song_id = sst.source_song_id;
Why is "source_song_title sst" in there? To the extent that
source_song_id is not unique, you are multiply updating rows
because of the self-join.
regards, tom lane
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Slow query
@ 2004-10-23 12:08 Joshua Marsh <icub3d@gmail.com>
parent: Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 0 replies; 112+ messages in thread
From: Joshua Marsh @ 2004-10-23 12:08 UTC (permalink / raw)
To: pgsql-performance
Any time you run subqueries, it's going to slow down the update
process a lot. Each record that is updated in source_song_title runs
two additional queries. When I do large updates like this, I usualy
Run a transaction that will select all the new data into a new table
on a join. For example
SELECT
a.*,
b.licensing_match_order,
b.affiliation_match_order,
d.title
INTO
updated_data
FROM
source_song_title AS a
INNER JOIN
source_system AS b
ON
b.id = d.id
INNER JOIN
source_song AS c
ON
a.id = c.id
INNER JOIN
source_title AS d
ON
a.id = d.id
I'm not sure that query does what you want, but you get the idea.
Then just drop the old table and rename the updated_data table. This
way instead of doing a bunch of updates, you do one select and a
rename.
-Josh
On Fri, 22 Oct 2004 16:37:14 -0400, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Roger Ging <rging@paccomsys.com> writes:
> > update source_song_title set
> > source_song_title_id = nextval('source_song_title_seq')
> > ,licensing_match_order = (select licensing_match_order from
> > source_system where source_system_id = ss.source_system_id)
> > ,affiliation_match_order = (select affiliation_match_order from
> > source_system where source_system_id = ss.source_system_id)
> > ,title = st.title
> > from source_song_title sst
> > join source_song ss on ss.source_song_id = sst.source_song_id
> > join source_title st on st.title_id = sst.title_id
> > where source_song_title.source_song_id = sst.source_song_id;
>
> Why is "source_song_title sst" in there? To the extent that
> source_song_id is not unique, you are multiply updating rows
> because of the self-join.
>
> regards, tom lane
>
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Don't 'kill -9' the postmaster
>
^ permalink raw reply [nested|flat] 112+ messages in thread
* Slow Query
@ 2005-07-14 00:06 Marc McIntyre <mmcintyre@squiz.net>
0 siblings, 1 reply; 112+ messages in thread
From: Marc McIntyre @ 2005-07-14 00:06 UTC (permalink / raw)
To: pgsql-performance
Hi,
I'm having a problem with a query that performs a sequential scan on a
table when it should be performing an index scan. The interesting thing
is, when we dumped the database on another server, it performed an index
scan on that server. The systems are running the same versions of
postgres (7.4.8) and the problem persists after running an "ANALYZE
VERBOSE" and after a "REINDEX TABLE sq_ast FORCE". The only difference
that i can see is that the postgresql.conf files differ slightly, and
the hardware is different. Note that the system performing the
sequential scan is a Dual 2.8GHz Xeon, 4GB Ram, 300GB HDD. And the
system performing an index scan is not as powerful.
A copy of the postgresql.conf for the system performing the index scan
can be found at http://beta.squiz.net/~mmcintyre/postgresql_squiz_uk.conf
A copy of the postgresql.conf for the system performing the sequential
scan can be found at http://beta.squiz.net/~mmcintyre/postgresql_future.conf
The Query:
SELECT a.assetid, a.short_name, a.type_code, a.status, l.linkid,
l.link_type, l.sort_order, lt.num_kids, u.url, ap.path,
CASE u.http
WHEN '1' THEN 'http'
WHEN '0' THEN 'https'
END AS protocol
FROM ((sq_ast a LEFT JOIN sq_ast_url u ON a.assetid = u.assetid) LEFT
JOIN sq_ast_path ap ON a.assetid = ap.assetid),sq_ast_lnk l,
sq_ast_lnk_tree lt WHERE a.assetid = l.minorid AND
l.linkid = lt.linkid AND l.majorid = '2' AND
l.link_type <= 2 ORDER BY sort_order
The EXPLAIN ANALYZE from the system performing an sequential scan:
QUERY PLAN
Sort (cost=30079.79..30079.89 rows=42 width=113) (actual time=39889.989..39890.346 rows=260 loops=1)
Sort Key: l.sort_order
-> Nested Loop (cost=25638.02..30078.65 rows=42 width=113) (actual time=9056.336..39888.557 rows=260 loops=1)
-> Merge Join (cost=25638.02..29736.01 rows=25 width=109) (actual time=9056.246..39389.359 rows=260 loops=1)
Merge Cond: (("outer".assetid)::text = "inner"."?column5?")
-> Merge Left Join (cost=25410.50..29132.82 rows=150816 width=97) (actual time=8378.176..38742.111 rows=150567 loops=1)
Merge Cond: (("outer".assetid)::text = ("inner".assetid)::text)
-> Merge Left Join (cost=25410.50..26165.14 rows=150816 width=83) (actual time=8378.130..9656.413 rows=150489 loops=1)
Merge Cond: ("outer"."?column5?" = "inner"."?column4?")
-> Sort (cost=25408.17..25785.21 rows=150816 width=48) (actual time=8377.733..8609.218 rows=150486 loops=1)
Sort Key: (a.assetid)::text
-> Seq Scan on sq_ast a (cost=0.00..12436.16 rows=150816 width=48) (actual time=0.011..5578.231 rows=151378 loops=1)
-> Sort (cost=2.33..2.43 rows=37 width=43) (actual time=0.364..0.428 rows=37 loops=1)
Sort Key: (u.assetid)::text
-> Seq Scan on sq_ast_url u (cost=0.00..1.37 rows=37 width=43) (actual time=0.023..0.161 rows=37 loops=1)
-> Index Scan using sq_ast_path_ast on sq_ast_path ap (cost=0.00..2016.98 rows=45893 width=23) (actual time=0.024..14041.571 rows=45812 loops=1)
-> Sort (cost=227.52..227.58 rows=25 width=21) (actual time=131.838..132.314 rows=260 loops=1)
Sort Key: (l.minorid)::text
-> Index Scan using sq_ast_lnk_majorid on sq_ast_lnk l (cost=0.00..226.94 rows=25 width=21) (actual time=0.169..126.201 rows=260 loops=1)
Index Cond: ((majorid)::text = '2'::text)
Filter: (link_type <= 2)
-> Index Scan using sq_ast_lnk_tree_linkid on sq_ast_lnk_tree lt (cost=0.00..13.66 rows=3 width=8) (actual time=1.539..1.900 rows=1 loops=260)
Index Cond: ("outer".linkid = lt.linkid)
Total runtime: 39930.395 ms
The EXPLAIN ANALYZE from the system performing an index scan scan:
Sort (cost=16873.64..16873.74 rows=40 width=113) (actual time=2169.905..2169.912 rows=13 loops=1)
Sort Key: l.sort_order
-> Nested Loop (cost=251.39..16872.58 rows=40 width=113) (actual time=45.724..2169.780 rows=13 loops=1)
-> Merge Join (cost=251.39..16506.42 rows=32 width=109) (actual time=45.561..2169.012 rows=13 loops=1)
Merge Cond: (("outer".assetid)::text = "inner"."?column5?")
-> Merge Left Join (cost=2.33..15881.92 rows=149982 width=97) (actual time=0.530..1948.718 rows=138569 loops=1)
Merge Cond: (("outer".assetid)::text = ("inner".assetid)::text)
-> Merge Left Join (cost=2.33..13056.04 rows=149982 width=83) (actual time=0.406..953.781 rows=138491 loops=1)
Merge Cond: (("outer".assetid)::text = "inner"."?column4?")
-> Index Scan using sq_ast_pkey on sq_ast a (cost=0.00..14952.78 rows=149982 width=48) (actual time=0.154..388.872 rows=138488 loops=1)
-> Sort (cost=2.33..2.43 rows=37 width=43) (actual time=0.235..0.264 rows=37 loops=1)
Sort Key: (u.assetid)::text
-> Seq Scan on sq_ast_url u (cost=0.00..1.37 rows=37 width=43) (actual time=0.036..0.103 rows=37 loops=1)
-> Index Scan using sq_ast_path_ast on sq_ast_path ap (cost=0.00..1926.18 rows=42071 width=23) (actual time=0.110..105.918 rows=42661 loops=1)
-> Sort (cost=249.05..249.14 rows=36 width=21) (actual time=0.310..0.324 rows=13 loops=1)
Sort Key: (l.minorid)::text
-> Index Scan using sq_ast_lnk_majorid on sq_ast_lnk l (cost=0.00..248.12 rows=36 width=21) (actual time=0.141..0.282 rows=13 loops=1)
Index Cond: ((majorid)::text = '2'::text)
Filter: (link_type <= 2)
-> Index Scan using sq_ast_lnk_tree_linkid on sq_ast_lnk_tree lt (cost=0.00..11.41 rows=2 width=8) (actual time=0.043..0.045 rows=1 loops=13)
Index Cond: ("outer".linkid = lt.linkid)
Total runtime: 2170.165 ms
(22 rows)
THE DESC of the sq_ast table.
future_v3_schema=# \d sq_ast
Table "public.sq_ast"
Column | Type | Modifiers
-----------------------+-----------------------------+---------------------------------------------
assetid | character varying(15) | not null
type_code | character varying(100) | not null
version | character varying(20) | not null default '0.0.0'::character
varying
name | character varying(255) | not null default ''::character varying
short_name | character varying(255) | not null default ''::character
varying
status | integer | not null default 1
languages | character varying(50) | not null default ''::character varying
charset | character varying(50) | not null default ''::character varying
force_secure | character(1) | not null default '0'::bpchar
created | timestamp without time zone | not null
created_userid | character varying(255) | not null
updated | timestamp without time zone | not null
updated_userid | character varying(255) | not null
published | timestamp without time zone |
published_userid | character varying(255) |
status_changed | timestamp without time zone |
status_changed_userid | character varying(255) |
Indexes:
"sq_asset_pkey" primary key, btree (assetid)
"sq_ast_created" btree (created)
"sq_ast_name" btree (name)
"sq_ast_published" btree (published)
"sq_ast_type_code" btree (type_code)
"sq_ast_updated" btree (updated)
Any ideas?
--
Marc McIntyre
MySource Matrix Lead Developer
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Slow Query
@ 2005-07-14 11:22 Ragnar Hafstað <gnari@simnet.is>
parent: Marc McIntyre <mmcintyre@squiz.net>
0 siblings, 1 reply; 112+ messages in thread
From: Ragnar Hafstað @ 2005-07-14 11:22 UTC (permalink / raw)
To: Marc McIntyre <mmcintyre@squiz.net>; +Cc: pgsql-performance
On Thu, 2005-07-14 at 10:06 +1000, Marc McIntyre wrote:
> I'm having a problem with a query that performs a sequential scan on a
> table when it should be performing an index scan. The interesting thing
> is, when we dumped the database on another server, it performed an index
> scan on that server.
...
> The EXPLAIN ANALYZE from the system performing an sequential scan:
>
> QUERY PLAN
> Sort (cost=30079.79..30079.89 rows=42 width=113) (actual time=39889.989..39890.346 rows=260 loops=1)
...
> The EXPLAIN ANALYZE from the system performing an index scan scan:
> Sort (cost=16873.64..16873.74 rows=40 width=113) (actual time=2169.905..2169.912 rows=13 loops=1)
looks like the first query is returning 260 rows,
but the second one 13
this may not be your problem, but are you sure you are using the same
query on the same data here ?
gnari
^ permalink raw reply [nested|flat] 112+ messages in thread
* Slow query
@ 2006-02-22 19:16 Jeremy Haile <jhaile@fastmail.fm>
0 siblings, 1 reply; 112+ messages in thread
From: Jeremy Haile @ 2006-02-22 19:16 UTC (permalink / raw)
To: pgsql-performance
I am running a query that joins against several large tables (~5 million
rows each). The query takes an exteremely long time to run, and the
explain output is a bit beyond my level of understanding. It is an
auto-generated query, so the aliases are fairly ugly. I can clean them
up (rename them) if it would help. Also, let me know if I can send any
more information that would help (e.g. table schema)
Also, is there any resources where I can get a better understanding of
what PostgreSQL means when it says "Sort" "Sort Key" "Bitmap Index Scan"
"Hash Cond" etc. etc. - and how to recognize problems by looking at the
output. I can understand the output for simple queries (e.g. is the
planner using an index or performing a seq. scan), but when you get to
more complex queries like the one below I lose my way =)
I would really appreciate it if someone from this list could tell me if
there is anything that is obviously wrong with the query or schema and
what I could do to improve the performance.
PostgreSQL 8.1
RedHat Enterprise Linux 4
--QUERY
select distinct city4_.region_id as region1_29_, city4_1_.name as
name29_, city4_.state_id as state2_30_
from registered_voters registered0_
inner join registered_voter_addresses addresses1_ on
registered0_.registered_voter_id=addresses1_.registered_voter_id
inner join registered_voter_addresses_regions regions2_ on
addresses1_.address_id=regions2_.registered_voter_addresses_address_id
inner join regions region3_ on
regions2_.regions_region_id=region3_.region_id
inner join cities city4_ on
addresses1_.city_id=city4_.region_id
inner join regions city4_1_ on
city4_.region_id=city4_1_.region_id
where region3_.region_id='093c44e8-f3b2-4c60-8be3-2b4d148f9f5a'
order by city4_1_.name
--EXPLAIN/ANALYZE OUTPUT
"Unique (cost=3572907.42..3623589.94 rows=4076438 width=93) (actual
time=2980825.714..3052333.753 rows=1124 loops=1)"
" -> Sort (cost=3572907.42..3585578.05 rows=5068252 width=93) (actual
time=2980825.710..2987407.888 rows=4918204 loops=1)"
" Sort Key: city4_1_.name, city4_.region_id, city4_.state_id"
" -> Hash Join (cost=717783.40..1430640.10 rows=5068252
width=93) (actual time=1400141.559..2016131.467 rows=4918204 loops=1)"
" Hash Cond:
(("outer".registered_voter_addresses_address_id)::text =
("inner".address_id)::text)"
" -> Bitmap Heap Scan on
registered_voter_addresses_regions regions2_ (cost=54794.95..575616.49
rows=5116843 width=80) (actual time=45814.469..155044.478 rows=4918205
loops=1)"
" Recheck Cond:
('093c44e8-f3b2-4c60-8be3-2b4d148f9f5a'::text =
(regions_region_id)::text)"
" -> Bitmap Index Scan on
reg_voter_address_region_region_idx (cost=0.00..54794.95 rows=5116843
width=0) (actual time=45807.157..45807.157 rows=4918205 loops=1)"
" Index Cond:
('093c44e8-f3b2-4c60-8be3-2b4d148f9f5a'::text =
(regions_region_id)::text)"
" -> Hash (cost=642308.89..642308.89 rows=741420
width=173) (actual time=1354217.934..1354217.934 rows=4918204 loops=1)"
" -> Hash Join (cost=328502.66..642308.89
rows=741420 width=173) (actual time=204565.031..1268303.832 rows=4918204
loops=1)"
" Hash Cond:
(("outer".registered_voter_id)::text =
("inner".registered_voter_id)::text)"
" -> Seq Scan on registered_voters
registered0_ (cost=0.00..173703.02 rows=4873202 width=40) (actual
time=0.005..39364.261 rows=4873167 loops=1)"
" -> Hash (cost=303970.34..303970.34
rows=748528 width=213) (actual time=204523.861..204523.861 rows=4918204
loops=1)"
" -> Hash Join (cost=263.22..303970.34
rows=748528 width=213) (actual time=101.628..140936.062 rows=4918204
loops=1)"
" Hash Cond:
(("outer".city_id)::text = ("inner".region_id)::text)"
" -> Seq Scan on
registered_voter_addresses addresses1_ (cost=0.00..271622.23
rows=4919923 width=120) (actual time=0.025..98416.667 rows=4918205
loops=1)"
" -> Hash (cost=260.35..260.35
rows=1147 width=173) (actual time=101.582..101.582 rows=1147 loops=1)"
" -> Hash Join
(cost=48.80..260.35 rows=1147 width=173) (actual time=88.608..98.984
rows=1147 loops=1)"
" Hash Cond:
(("outer".region_id)::text = ("inner".region_id)::text)"
" -> Seq Scan on
regions city4_1_ (cost=0.00..162.39 rows=7539 width=53) (actual
time=0.048..35.204 rows=7539 loops=1)"
" -> Hash
(cost=45.93..45.93 rows=1147 width=120) (actual time=48.896..48.896
rows=1147 loops=1)"
" -> Nested Loop
(cost=0.00..45.93 rows=1147 width=120) (actual time=35.791..47.012
rows=1147 loops=1)"
" -> Index
Scan using regions_pkey on regions region3_ (cost=0.00..5.99 rows=1
width=40) (actual time=35.761..35.763 rows=1 loops=1)"
"
Index Cond: ((region_id)::text =
'093c44e8-f3b2-4c60-8be3-2b4d148f9f5a'::text)"
" -> Seq
Scan on cities city4_ (cost=0.00..28.47 rows=1147 width=80) (actual
time=0.022..9.476 rows=1147 loops=1)"
"Total runtime: 3052707.269 ms"
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Slow query
@ 2006-02-23 17:25 Tom Lane <tgl@sss.pgh.pa.us>
parent: Jeremy Haile <jhaile@fastmail.fm>
0 siblings, 0 replies; 112+ messages in thread
From: Tom Lane @ 2006-02-23 17:25 UTC (permalink / raw)
To: Jeremy Haile <jhaile@fastmail.fm>; +Cc: pgsql-performance
"Jeremy Haile" <jhaile@fastmail.fm> writes:
> I am running a query that joins against several large tables (~5 million
> rows each). The query takes an exteremely long time to run, and the
> explain output is a bit beyond my level of understanding. It is an
> auto-generated query, so the aliases are fairly ugly.
Yah :-(
> select distinct city4_.region_id as region1_29_, city4_1_.name as
> name29_, city4_.state_id as state2_30_
> from registered_voters registered0_
> inner join registered_voter_addresses addresses1_ on
> registered0_.registered_voter_id=addresses1_.registered_voter_id
> inner join registered_voter_addresses_regions regions2_ on
> addresses1_.address_id=regions2_.registered_voter_addresses_address_id
> inner join regions region3_ on
> regions2_.regions_region_id=region3_.region_id
> inner join cities city4_ on
> addresses1_.city_id=city4_.region_id
> inner join regions city4_1_ on
> city4_.region_id=city4_1_.region_id
> where region3_.region_id='093c44e8-f3b2-4c60-8be3-2b4d148f9f5a'
> order by city4_1_.name
AFAICS the planner is doing about the best you can hope the machine to
do --- it's not making any serious estimation errors, and the plan is
pretty reasonable for the given query. The problem is that you are
forming a very large join result (4918204 rows) and then doing a
DISTINCT that reduces this to only 1124 rows ... but the damage of
computing that huge join has already been done. The machine is not
going to be able to think its way out of this one --- it's up to you
to think of a better formulation of the query.
Offhand I'd try something involving joining just city4_/city4_1_
(which should not need DISTINCT, I think) and then using WHERE
EXISTS(SELECT ... FROM the-other-tables) to filter out the cities
you don't want. The reason this can be a win is that the EXISTS
formulation will stop running the sub-select as soon as it's produced a
single row for the current city, rather than generating thousands of
similar rows that will be thrown away by DISTINCT as you have here.
This assumes that the fraction of cities passing the query is
substantial, as it appears from the rowcounts in your EXPLAIN output.
If only a tiny fraction of them passed, then the time wasted in failing
EXISTS probes might eat up the savings.
regards, tom lane
^ permalink raw reply [nested|flat] 112+ messages in thread
* Slow Query
@ 2006-11-21 17:03 Joe Lester <joe_lester@sweetwater.com>
0 siblings, 0 replies; 112+ messages in thread
From: Joe Lester @ 2006-11-21 17:03 UTC (permalink / raw)
To: pgsql-performance
I have a commonly run query that has been executing fine for the last
year or so. It usually completes in under 1 second. However, in the
past week or so the performance of the query has become erratic,
taking anywhere from 100 ms to 100,000 ms to complete the same query
(executed just seconds apart). It's the holiday season, so there has
probably been an increase in server activity.
SELECT products.* FROM products LEFT JOIN product_identifiers ON
product_identifiers.product_id = products.id WHERE
product_identifiers.identifier = '21A40606099800168' OR
products.part_number = '21A40606099800168';
I'm just using this query as a concrete example. I have the same
problem with other queries (I suspect most queries). Here's the
EXPLAIN ANALYZE output run twice. The first time is fast. The second
time is slow.
Then, below that, I've attached some specs from my configuration...
if anyone sees anything that is out of whack... I'm new to
troubleshooting this sort of thing, so any advise would be appreciated.
-------
officelink=# EXPLAIN ANALYZE SELECT products.* FROM products LEFT
JOIN product_identifiers ON product_identifiers.product_id =
products.id WHERE product_identifiers.identifier =
'21A40606099800168' OR products.part_number = '21A40606099800168';
QUERY PLAN
------------------------------------------------------------------------
------------------------------------------------------------------------
------------------------
Merge Left Join (cost=0.00..4264.09 rows=40368 width=107) (actual
time=755.150..755.150 rows=0 loops=1)
Merge Cond: ("outer".id = "inner".product_id)
Filter: ((("inner".identifier)::text = '21A40606099800168'::text)
OR (("outer".part_number)::text = '21A40606099800168'::text))
-> Index Scan using products_id_idx on products
(cost=0.00..3680.34 rows=40368 width=107) (actual time=8.762..643.550
rows=40382 loops=1)
-> Index Scan using product_identifiers_product_id_idx on
product_identifiers (cost=0.00..368.69 rows=6524 width=20) (actual
time=0.131..76.958 rows=6532 loops=1)
Total runtime: 755.301 ms
(6 rows)
officelink=# EXPLAIN ANALYZE SELECT products.* FROM products LEFT
JOIN product_identifiers ON product_identifiers.product_id =
products.id WHERE product_identifiers.identifier =
'21A40606099800168' OR products.part_number = '21A40606099800168';
QUERY PLAN
------------------------------------------------------------------------
------------------------------------------------------------------------
--------------------------
Merge Left Join (cost=0.00..4264.09 rows=40368 width=107) (actual
time=25885.235..25885.235 rows=0 loops=1)
Merge Cond: ("outer".id = "inner".product_id)
Filter: ((("inner".identifier)::text = '21A40606099800168'::text)
OR (("outer".part_number)::text = '21A40606099800168'::text))
-> Index Scan using products_id_idx on products
(cost=0.00..3680.34 rows=40368 width=107) (actual
time=0.070..23503.630 rows=40382 loops=1)
-> Index Scan using product_identifiers_product_id_idx on
product_identifiers (cost=0.00..368.69 rows=6524 width=20) (actual
time=0.058..2346.662 rows=6532 loops=1)
Total runtime: 25885.375 ms
(6 rows)
Server Specs:
Intel Core Solo Mac Mini running OS 10.4.7
1.25 GB RAM
30 GB of space left on the 55 GB internal hard drive
Usage:
400 persistent connections from various clients
top usually sits at 85%-95% idle.
postgresql.conf Settings [non-defualt]:
max_connections = 500
shared_buffers = 10000
work_mem = 2048
max_fsm_pages = 150000
max_stack_depth = 6000
archive_command = 'cp -i %p /Volumes/Backup/wal_archive/%f </dev/null'
effective_cache_size = 30000
log_min_duration_statement = 2000
log_line_prefix = '%t %h '
stats_start_collector = on
stats_row_level = on
autovacuum = on
autovacuum_naptime = 60
autovacuum_vacuum_threshold = 150
autovacuum_vacuum_scale_factor = 0.00000001
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Slow Query
@ 2006-11-21 19:52 Joe Lester <joe_lester@sweetwater.com>
parent: Ragnar Hafstað <gnari@simnet.is>
0 siblings, 0 replies; 112+ messages in thread
From: Joe Lester @ 2006-11-21 19:52 UTC (permalink / raw)
To: pgsql-performance
Pretty sure. When I ran the queries I was in psql. I ran the first
query. Then I pressed the up arrow to re-run the same query. I did
not notice that about the rows. Might reindexing be in order?
On Jul 14, 2005, at 7:22 AM, Ragnar Hafstað wrote:
> On Thu, 2005-07-14 at 10:06 +1000, Marc McIntyre wrote:
>
>> I'm having a problem with a query that performs a sequential scan
>> on a
>> table when it should be performing an index scan. The interesting
>> thing
>> is, when we dumped the database on another server, it performed an
>> index
>> scan on that server.
> ...
>> The EXPLAIN ANALYZE from the system performing an sequential scan:
>>
>> QUERY PLAN
>> Sort (cost=30079.79..30079.89 rows=42 width=113) (actual
>> time=39889.989..39890.346 rows=260 loops=1)
> ...
>> The EXPLAIN ANALYZE from the system performing an index scan scan:
>> Sort (cost=16873.64..16873.74 rows=40 width=113) (actual
>> time=2169.905..2169.912 rows=13 loops=1)
>
> looks like the first query is returning 260 rows,
> but the second one 13
>
> this may not be your problem, but are you sure you are using the same
> query on the same data here ?
>
> gnari
>
>
>
> ---------------------------(end of
> broadcast)---------------------------
> TIP 2: Don't 'kill -9' the postmaster
>
>
^ permalink raw reply [nested|flat] 112+ messages in thread
* slow query
@ 2007-07-02 10:07 Vidhya Bondre <meetvbondre@gmail.com>
0 siblings, 1 reply; 112+ messages in thread
From: Vidhya Bondre @ 2007-07-02 10:07 UTC (permalink / raw)
To: pgsql-performance
Hi all,
I need a very urgent help from you all in below case.
I have a query
SELECT amp.campaign_id, dam.allocation_map_id,amp.optimize_type,
amp.optimize_by_days, amp.rate, amp.action_id,amp.actions_delta,
amp.vearned_today, amp.creative_id, amp.channel_code,SUM(CASE
dam.sqldatewhen 20070701 then
dam.actions_delivered else 0 end) as action_yest,SUM(CASE sign(20070624 -
dam.sqldate) when -1 then dam.actions_delivered else 0 end) as
action_wk1,SUM(CASE sign(20070617 - dam.sqldate) when -1 then
dam.actions_delivered else 0 end) as action_wk2,SUM(CASE sign(20070610 -
dam.sqldate) when -1 then dam.actions_delivered else 0 end) as
action_wk3,SUM(CASE sign(20070603 - dam.sqldate) when -1 then
dam.actions_delivered else 0 end) as action_wk4,SUM(CASE sign(20070527 -
dam.sqldate) when -1 then dam.actions_delivered else 0 end) as
action_wk5,SUM(CASE sign(20070520 - dam.sqldate) when -1 then
dam.actions_delivered else 0 end) as action_wk6,SUM(CASE sign(20070513 -
dam.sqldate) when -1 then dam.actions_delivered else 0 end) as
action_wk7,SUM(CASE sign(20070506 - dam.sqldate) when -1 then
dam.actions_delivered else 0 end) as action_wk8,SUM(CASE dam.sqldate when
20070701 then dam.vearned_total else 0 end) as earned_yest,SUM(CASE
sign(20070624 - dam.sqldate) when -1 then dam.vearned_total else 0 end) as
vearned_wk1,SUM(CASE sign(20070617 - dam.sqldate) when -1 then
dam.vearned_total else 0 end) as vearned_wk2,SUM(CASE sign(20070610 -
dam.sqldate) when -1 then dam.vearned_total else 0 end) as
vearned_wk3,SUM(CASE sign(20070603 - dam.sqldate) when -1 then
dam.vearned_total else 0 end) as vearned_wk4,SUM(CASE sign(20070527 -
dam.sqldate) when -1 then dam.vearned_total else 0 end) as
vearned_wk5,SUM(CASE sign(20070520 - dam.sqldate) when -1 then
dam.vearned_total else 0 end) as vearned_wk6,SUM(CASE sign(20070513 -
dam.sqldate) when -1 then dam.vearned_total else 0 end) as
vearned_wk7,SUM(CASE sign(20070506 - dam.sqldate) when -1 then
dam.vearned_total else 0 end) as vearned_wk8,SUM(CASE dam.sqldate when
20070701 then dam.vactions_delivered else 0 end) as vactions_yest,SUM(CASE
sign(20070624 - dam.sqldate) when -1 then dam.vactions_delivered else 0 end)
as vactionsdel1,SUM(CASE sign(20070617 - dam.sqldate ) when -1 then
dam.vactions_delivered else 0 end) as vactionsdel2,SUM(CASE sign(20070610 -
dam.sqldate) when -1 then dam.vactions_delivered else 0 end) as
vactionsdel3,SUM(CASE sign(20070603 - dam.sqldate) when -1 then
dam.vactions_delivered else 0 end) as vactionsdel4,SUM(CASE sign(20070527 -
dam.sqldate) when -1 then dam.vactions_delivered else 0 end) as
vactionsdel5, SUM(CASE sign(20070520 - dam.sqldate) when -1 then
dam.vactions_delivered else 0 end) as vactionsdel6,SUM(CASE sign(20070513 -
dam.sqldate) when -1 then dam.vactions_delivered else 0 end) as
vactionsdel7,SUM(CASE sign(20070506 - dam.sqldate) when -1 then
dam.vactions_delivered else 0 end) as vactionsdel8 FROM delivered_action_map
dam INNER JOIN (SELECT a.campaign_id, a.optimize_type,a.optimize_by_days,
a.rate, a.action_id, am.creative_id, am.channel_code, amt.actions_delta,
amt.vearned_today, am.id AS allocation_map_id FROM (SELECT c.campaign_id ,
c.optimize_type, c.optimize_by_days, a1.rate, a1.id AS action_id FROM action
a1 INNER JOIN (SELECT c1.asset_id AS campaign_id, ca.value AS
optimize_type,c1.optimize_by_days AS optimize_by_days FROM campaign c1 INNER
JOIN (SELECT ca2.campaign_id AS campaign_id, ca3.value AS value FROM
campaign_attributes ca2, campaign_attributes ca3 WHERE ca2.campaign_id =
ca3.campaign_id AND ca2.attribute='OPTIMIZE_STATUS' AND ca2.value = '1'AND
ca3. attribute ='OPTIMIZE_TYPE') as ca ON c1.asset_id=ca.campaign_id AND
20070702 BETWEEN (c1.start_date - interval '1 day') AND
(c1.end_date+interval '1day') AND
c1.status = 'A' AND c1.revenue_type != 'FOC' AND c1.action_type >= 1 AND
c1.optimize_by_days > 0) AS c ON a1.campaign_id = c.campaign_id AND
a1.status = 'A') AS a, allocation_map am, action_metrics amt WHERE
a.action_id = amt.action_id AND am.id = amt.allocation_map_id AND
am.status= 'A') AS amp ON
dam.allocation_map_id= amp.allocation_map_id AND dam.action_id =
amp.action_id GROUP BY amp.campaign_id, amp.optimize_type,
amp.optimize_by_days, amp.rate, amp.action_id, amp.actions_delta ,
amp.creative_id, amp.channel_code, dam.allocation_map_id, amp.vearned_today;
after vacuuming the db it has become very very slow ... 100 times slow.
Please suggest ?
Regards
Vidhya
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: slow query
@ 2007-07-02 13:20 Nis Jørgensen <nis@superlativ.dk>
parent: Vidhya Bondre <meetvbondre@gmail.com>
0 siblings, 0 replies; 112+ messages in thread
From: Nis Jørgensen @ 2007-07-02 13:20 UTC (permalink / raw)
To: pgsql-performance
Vidhya Bondre skrev:
> Hi all,
>
> I need a very urgent help from you all in below case.
>
> I have a query
[snipped]
> after vacuuming the db it has become very very slow ... 100 times slow.
>
> Please suggest ?
Suggestions for getting more/better responses:
- Format your query nicely before posting it.
- Post the relevant table definitions, including indices
- Tell us what the query is supposed to do.
Suggestions for finding the cause of your problem:
- Run "EXPLAIN ANALYZE" on the query.
- Try to "remove bits" of the query to see which bits slow it down - try
to find a "minimal query" which shows the performance problem. If you
can, use the output of "EXPLAIN ANALYZE" obtained above. For instance,
all the SUMs in the SELECT clause are unlikely to significantly affect
the running time.
- Run "EXPLAIN ANALYZE" on the "minimal query", post the results.
Nis
^ permalink raw reply [nested|flat] 112+ messages in thread
* Slow Query
@ 2007-09-01 17:29 Shawn <postgres@xmtservices.net>
0 siblings, 2 replies; 112+ messages in thread
From: Shawn @ 2007-09-01 17:29 UTC (permalink / raw)
To: pgsql-performance
Hello,
I have a recurring script that updates some tables from an MS_SQL
server. One of the operations sets a field in all records to null in
preparation of being updated with values from the other server. The
SQL statement is:
update shawns_data set alias = null;
Alias is a type varchar(8)
The table has 26 fields per record and there are about 15,700
records. The server hardware is a dual QUAD-CORE Intel 2 GHz XEON dell
2950 server with 4 drive SAS RAID-5 array, and 16G of RAM. The OS is
Slackware 11 with some updatews and Postgres v8.2.4 built from source.
Even after VACUUM this simple line takes 35 sec to complete. Other
more complicated deletes and updates, some of the tables in this
database are over 300 million records, take as much time as this
simple query.
My question: Is there a better, ie. faster, way to do this task?
Shawn
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Slow Query
@ 2007-09-01 18:09 Tom Lane <tgl@sss.pgh.pa.us>
parent: Shawn <postgres@xmtservices.net>
1 sibling, 1 reply; 112+ messages in thread
From: Tom Lane @ 2007-09-01 18:09 UTC (permalink / raw)
To: Shawn <postgres@xmtservices.net>; +Cc: pgsql-performance
Shawn <postgres@xmtservices.net> writes:
> update shawns_data set alias = null;
> Alias is a type varchar(8)
> The table has 26 fields per record and there are about 15,700
> records. The server hardware is a dual QUAD-CORE Intel 2 GHz XEON dell
> 2950 server with 4 drive SAS RAID-5 array, and 16G of RAM. The OS is
> Slackware 11 with some updatews and Postgres v8.2.4 built from source.
> Even after VACUUM this simple line takes 35 sec to complete.
Seems like a lot. Table bloat maybe (what does VACUUM VERBOSE say about
this table)? An unreasonably large number of indexes to update?
Foreign key checks? (Though unless you have FKs pointing at alias,
I'd think 8.2 would avoid needing to make any FK checks.)
Could we see EXPLAIN ANALYZE output for this operation? (If you don't
really want to zap the column right now, wrap the EXPLAIN in
BEGIN/ROLLBACK.)
regards, tom lane
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Slow Query
@ 2007-09-01 20:18 Shawn <postgres@xmtservices.net>
parent: Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 112+ messages in thread
From: Shawn @ 2007-09-01 20:18 UTC (permalink / raw)
To: pgsql-performance
Hi Tom,
Thanks for replying.
There are no FK's, indexes, or dependents on the alias field.
The system is in the middle of its weekly full activity log resync,
about 600 Million records. It will be done later this evening and I
will run the explain analyze thenand I will post the results. I will
also run a vacuum full analyze on it before the run and have timing on.
Shawn
On Sat, 01 Sep 2007 14:09:54 -0400 Tom Lane
<tgl@sss.pgh.pa.us> wrote:
> Shawn <postgres@xmtservices.net> writes:
> > update shawns_data set alias = null;
>
> > Alias is a type varchar(8)
>
> > The table has 26 fields per record and there are about 15,700
> > records. The server hardware is a dual QUAD-CORE Intel 2 GHz XEON
> > dell 2950 server with 4 drive SAS RAID-5 array, and 16G of RAM.
> > The OS is Slackware 11 with some updatews and Postgres v8.2.4 built
> > from source.
>
> > Even after VACUUM this simple line takes 35 sec to complete.
>
> Seems like a lot. Table bloat maybe (what does VACUUM VERBOSE say
> about this table)? An unreasonably large number of indexes to update?
> Foreign key checks? (Though unless you have FKs pointing at alias,
> I'd think 8.2 would avoid needing to make any FK checks.)
>
> Could we see EXPLAIN ANALYZE output for this operation? (If you don't
> really want to zap the column right now, wrap the EXPLAIN in
> BEGIN/ROLLBACK.)
>
> regards, tom lane
>
> ---------------------------(end of
> broadcast)--------------------------- TIP 6: explain analyze is your
> friend
>
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Slow Query
@ 2007-09-02 00:35 Shawn <postgres@xmtservices.net>
parent: Shawn <postgres@xmtservices.net>
0 siblings, 1 reply; 112+ messages in thread
From: Shawn @ 2007-09-02 00:35 UTC (permalink / raw)
To: pgsql-performance
Ok,
The query just ran and here is the basic output:
UPDATE 15445
Time: 22121.141 ms
and
explain ANALYZE update shawns_data set alias = null;
QUERY PLAN
-----------------------------------------------------------------------------------------------------------------
Seq Scan on shawns_data (cost=0.00..465.45 rows=15445 width=480) (actual time=0.034..67.743 rows=15445 loops=1)
Total runtime: 1865.002 ms
(2 rows)
Shawn
On Sat, 1 Sep 2007 13:18:16 -0700
Shawn <postgres@xmtservices.net> wrote:
>
> Hi Tom,
>
> Thanks for replying.
>
> There are no FK's, indexes, or dependents on the alias field.
>
> The system is in the middle of its weekly full activity log resync,
> about 600 Million records. It will be done later this evening and I
> will run the explain analyze thenand I will post the results. I will
> also run a vacuum full analyze on it before the run and have timing
> on.
>
> Shawn
>
>
> On Sat, 01 Sep 2007 14:09:54 -0400 Tom Lane
> <tgl@sss.pgh.pa.us> wrote:
>
> > Shawn <postgres@xmtservices.net> writes:
> > > update shawns_data set alias = null;
> >
> > > Alias is a type varchar(8)
> >
> > > The table has 26 fields per record and there are about 15,700
> > > records. The server hardware is a dual QUAD-CORE Intel 2 GHz XEON
> > > dell 2950 server with 4 drive SAS RAID-5 array, and 16G of RAM.
> > > The OS is Slackware 11 with some updatews and Postgres v8.2.4
> > > built from source.
> >
> > > Even after VACUUM this simple line takes 35 sec to complete.
> >
> > Seems like a lot. Table bloat maybe (what does VACUUM VERBOSE say
> > about this table)? An unreasonably large number of indexes to
> > update? Foreign key checks? (Though unless you have FKs pointing
> > at alias, I'd think 8.2 would avoid needing to make any FK checks.)
> >
> > Could we see EXPLAIN ANALYZE output for this operation? (If you
> > don't really want to zap the column right now, wrap the EXPLAIN in
> > BEGIN/ROLLBACK.)
> >
> > regards, tom lane
> >
> > ---------------------------(end of
> > broadcast)--------------------------- TIP 6: explain analyze is your
> > friend
> >
>
> ---------------------------(end of
> broadcast)--------------------------- TIP 1: if posting/reading
> through Usenet, please send an appropriate subscribe-nomail command
> to majordomo@postgresql.org so that your message can get through to
> the mailing list cleanly
>
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Slow Query
@ 2007-09-02 03:00 Tom Lane <tgl@sss.pgh.pa.us>
parent: Shawn <postgres@xmtservices.net>
0 siblings, 1 reply; 112+ messages in thread
From: Tom Lane @ 2007-09-02 03:00 UTC (permalink / raw)
To: Shawn <postgres@xmtservices.net>; +Cc: pgsql-performance
Shawn <postgres@xmtservices.net> writes:
> The query just ran and here is the basic output:
> UPDATE 15445
> Time: 22121.141 ms
> and
> explain ANALYZE update shawns_data set alias = null;
> QUERY PLAN
> -----------------------------------------------------------------------------------------------------------------
> Seq Scan on shawns_data (cost=0.00..465.45 rows=15445 width=480) (actual time=0.034..67.743 rows=15445 loops=1)
> Total runtime: 1865.002 ms
> (2 rows)
Hmmm ... did you run the real query and the EXPLAIN in immediate
succession? If so, the only reason I can think of for the speed
difference is that all the rows were fetched already for the second
run. Which doesn't make a lot of sense given the hardware specs
you mentioned. Try watching "vmstat 1" and see if there's some
noticeable difference in the behavior.
regards, tom lane
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Slow Query
@ 2007-09-02 15:49 Kevin Grittner <Kevin.Grittner@wicourts.gov>
parent: Shawn <postgres@xmtservices.net>
1 sibling, 1 reply; 112+ messages in thread
From: Kevin Grittner @ 2007-09-02 15:49 UTC (permalink / raw)
To: pgsql-performance; Shawn <postgres@xmtservices.net>
>>> On Sat, Sep 1, 2007 at 12:29 PM, in message
<20070901102947.0c0a50a3@boffin.xmtservices.net>, Shawn
<postgres@xmtservices.net> wrote:
> update shawns_data set alias = null;
> Even after VACUUM this simple line takes 35 sec to complete.
Would any rows already have a null alias when you run this?
If so, try adding 'where alias is not null' to the query.
-Kevin
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Slow Query
@ 2007-09-03 16:15 Shawn <postgres@xmtservices.net>
parent: Kevin Grittner <Kevin.Grittner@wicourts.gov>
0 siblings, 1 reply; 112+ messages in thread
From: Shawn @ 2007-09-03 16:15 UTC (permalink / raw)
To: pgsql-performance
Thanks Kevin,
This one initially added about 10sec to the run but I added a HASH
index on the alias field and its now about 5 sec average runtime, a net
improvement.
Shawn
On Sun, 02 Sep 2007 10:49:09 -0500 "Kevin Grittner"
<Kevin.Grittner@wicourts.gov> wrote:
> >>> On Sat, Sep 1, 2007 at 12:29 PM, in message
> <20070901102947.0c0a50a3@boffin.xmtservices.net>, Shawn
> <postgres@xmtservices.net> wrote:
> > update shawns_data set alias = null;
> > Even after VACUUM this simple line takes 35 sec to complete.
>
> Would any rows already have a null alias when you run this?
>
> If so, try adding 'where alias is not null' to the query.
>
> -Kevin
>
>
>
>
> ---------------------------(end of
> broadcast)--------------------------- TIP 2: Don't 'kill -9' the
> postmaster
>
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Slow Query
@ 2007-09-03 16:57 Shawn <postgres@xmtservices.net>
parent: Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 112+ messages in thread
From: Shawn @ 2007-09-03 16:57 UTC (permalink / raw)
To: pgsql-performance
On Sat, 01 Sep 2007 23:00:10 -0400
Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Shawn <postgres@xmtservices.net> writes:
> > The query just ran and here is the basic output:
>
> > UPDATE 15445
> > Time: 22121.141 ms
>
> > and
>
> > explain ANALYZE update shawns_data set alias = null;
> > QUERY
> > PLAN
> > -----------------------------------------------------------------------------------------------------------------
> > Seq Scan on shawns_data (cost=0.00..465.45 rows=15445 width=480)
> > (actual time=0.034..67.743 rows=15445 loops=1) Total runtime:
> > 1865.002 ms (2 rows)
>
> Hmmm ... did you run the real query and the EXPLAIN in immediate
> succession? If so, the only reason I can think of for the speed
> difference is that all the rows were fetched already for the second
> run. Which doesn't make a lot of sense given the hardware specs
> you mentioned. Try watching "vmstat 1" and see if there's some
> noticeable difference in the behavior.
>
Actually no,
The runs were at least an 24 hours apart.
Running the query standalone was quite interesting. There's a lot of
disk output:
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 2580 450992 84420 16007864 0 0 1 1 1 2 3 0 94 2
0 0 2580 450992 84420 16007864 0 0 0 0 1003 18 0 0 100 0
0 0 2580 450660 84420 16007864 0 0 0 6764 1127 202 0 0 100 0
0 0 2580 450660 84420 16007864 0 0 0 0 1004 28 0 0 100 0
0 0 2580 450660 84420 16007864 0 0 0 0 1011 58 0 0 100 0
0 0 2580 450660 84420 16007864 0 0 0 0 1011 50 0 0 100 0
0 0 2580 450660 84428 16007864 0 0 0 32 1011 53 0 0 100 0
0 0 2580 450660 84428 16007864 0 0 0 3034 1095 191 0 0 100 0
0 0 2580 450164 84428 16007864 0 0 0 0 1032 362 0 0 100 0
0 0 2580 450164 84428 16007864 0 0 0 0 1002 22 0 0 100 0
1 0 2580 446444 84428 16012240 0 0 0 2568 1024 398 4 1 95 0
1 0 2580 456172 84428 16001760 0 0 8 11549 1136 616 10 3 87 0
0 0 2580 452164 84428 16005524 0 0 4 26580 1885 1479 4 2 93 1
0 0 2580 452164 84436 16005516 0 0 0 40 1004 26 0 0 100 0
0 0 2580 452164 84436 16005524 0 0 0 0 1001 36 0 0 100 0
0 0 2580 452164 84436 16005524 0 0 0 0 1001 24 0 0 100 0
0 0 2580 452164 84436 16005524 0 0 0 0 1001 36 0 0 100 0
0 0 2580 452164 84436 16005524 0 0 0 3 1005 27 0 0 100 0
0 0 2580 452040 84436 16005524 0 0 0 0 1006 41 0 0 100 0
0 0 2580 452040 84436 16005524 0 0 0 0 1002 20 0 0 100 0
Also it runs a lot faster by itself, so I am thinking there is some query early on that is
really fragmenting the table. Here's the whole script:
\timing
select now();
create temporary table unit_operation(lid varchar(5), hang_time varchar(3),multi_csd_enabled bool,multi_node_enabled bool,wide_area_enabled bool);
create temporary table unit(lid varchar(5),alias char(8),type smallint);
\copy unit from '/tmp/wc_unit'
\copy unit_operation from '/tmp/wc_unit_operation'
truncate csd;
insert into csd(lid,alias,type) select lid,alias,type from unit;
update csd set hang_time = uo.hang_time, multi_csd_enabled = uo.multi_csd_enabled, multi_node_enabled = uo.multi_node_enabled, wide_area_enabled = uo.wide_area_enabled from unit_operation uo where csd.lid = uo.lid;
create temporary table groups(lid varchar(5),alias varchar(8));
create temporary table group_operation(lid varchar(15),hang_time varchar(8));
\copy groups from '/tmp/wc_groups'
\copy group_operation from '/tmp/wc_group_operation'
truncate gid;
insert into gid(gid,alias) select lid,alias from groups;
update gid set hang_time = go.hang_time from group_operation go where gid.gid = go.lid;
truncate rc_unit_state;
\copy rc_unit_state from '/tmp/wc_rc_unit_state' with null as ''
update lidrange set csd = false, data = false, disable = false, lost = false;
update lidrange set csd = true from csd where lidrange.lid = csd.lid;
update lidrange set data = true from shawns_data where lidrange.lid = shawns_data.lid;
update lidrange set disable = true from rc_unit_state where lidrange.lid = rc_unit_state.lid;
update shawns_data set alias = null where alias is not null;
update shawns_data set alias = csd.alias from csd where shawns_data.lid = csd.lid;
update lidrange set lost = true from shawns_data where shawns_data.lid = lidrange.lid and shawns_data.lost = true
\i '/home/shawn/vacuum_data.sql'
As you can see there is nothing really accessing the shawns_data table before the query in question.
I have also allocated 380 Megs of RAM to any virtual process, I can't see where any of these tables
could take up this much RAM. Here is my postgresql.conf:
# -----------------------------
# PostgreSQL configuration file
# -----------------------------
#
# This file consists of lines of the form:
#
# name = value
#
# (The '=' is optional.) White space may be used. Comments are introduced
# with '#' anywhere on a line. The complete list of option names and
# allowed values can be found in the PostgreSQL documentation. The
# commented-out settings shown in this file represent the default values.
#
# Please note that re-commenting a setting is NOT sufficient to revert it
# to the default value, unless you restart the server.
#
# Any option can also be given as a command line switch to the server,
# e.g., 'postgres -c log_connections=on'. Some options can be changed at
# run-time with the 'SET' SQL command.
#
# This file is read on server startup and when the server receives a
# SIGHUP. If you edit the file on a running system, you have to SIGHUP the
# server for the changes to take effect, or use "pg_ctl reload". Some
# settings, which are marked below, require a server shutdown and restart
# to take effect.
#
# Memory units: kB = kilobytes MB = megabytes GB = gigabytes
# Time units: ms = milliseconds s = seconds min = minutes h = hours d = days
#---------------------------------------------------------------------------
# FILE LOCATIONS
#---------------------------------------------------------------------------
# The default values of these variables are driven from the -D command line
# switch or PGDATA environment variable, represented here as ConfigDir.
#data_directory = 'ConfigDir' # use data in another directory
# (change requires restart)
#hba_file = 'ConfigDir/pg_hba.conf' # host-based authentication file
# (change requires restart)
#ident_file = 'ConfigDir/pg_ident.conf' # ident configuration file
# (change requires restart)
# If external_pid_file is not explicitly set, no extra PID file is written.
#external_pid_file = '(none)' # write an extra PID file
# (change requires restart)
#---------------------------------------------------------------------------
# CONNECTIONS AND AUTHENTICATION
#---------------------------------------------------------------------------
# - Connection Settings -
#listen_addresses = 'localhost' # what IP address(es) to listen on;
# comma-separated list of addresses;
# defaults to 'localhost', '*' = all
# (change requires restart)
#port = 5432 # (change requires restart)
max_connections = 20 # (change requires restart)
# Note: increasing max_connections costs ~400 bytes of shared memory per
# connection slot, plus lock space (see max_locks_per_transaction). You
# might also need to raise shared_buffers to support more connections.
#superuser_reserved_connections = 3 # (change requires restart)
#unix_socket_directory = '' # (change requires restart)
#unix_socket_group = '' # (change requires restart)
#unix_socket_permissions = 0777 # octal
# (change requires restart)
#bonjour_name = '' # defaults to the computer name
# (change requires restart)
# - Security & Authentication -
#authentication_timeout = 1min # 1s-600s
#ssl = off # (change requires restart)
#password_encryption = on
#db_user_namespace = off
# Kerberos
#krb_server_keyfile = '' # (change requires restart)
#krb_srvname = 'postgres' # (change requires restart)
#krb_server_hostname = '' # empty string matches any keytab entry
# (change requires restart)
#krb_caseins_users = off # (change requires restart)
# - TCP Keepalives -
# see 'man 7 tcp' for details
#tcp_keepalives_idle = 0 # TCP_KEEPIDLE, in seconds;
# 0 selects the system default
#tcp_keepalives_interval = 0 # TCP_KEEPINTVL, in seconds;
# 0 selects the system default
#tcp_keepalives_count = 0 # TCP_KEEPCNT;
# 0 selects the system default
#---------------------------------------------------------------------------
# RESOURCE USAGE (except WAL)
#---------------------------------------------------------------------------
# - Memory -
shared_buffers = 300MB # min 128kB or max_connections*16kB
# (change requires restart)
#temp_buffers = 8MB # min 800kB
#max_prepared_transactions = 5 # can be 0 or more
# (change requires restart)
# Note: increasing max_prepared_transactions costs ~600 bytes of shared memory
# per transaction slot, plus lock space (see max_locks_per_transaction).
work_mem = 300MB # min 64kB
maintenance_work_mem = 260MB # min 1MB
max_stack_depth = 7MB # min 100kB
# - Free Space Map -
max_fsm_pages = 15360000 # min max_fsm_relations*16, 6 bytes each
# (change requires restart)
#max_fsm_relations = 1000 # min 100, ~70 bytes each
# (change requires restart)
# - Kernel Resource Usage -
#max_files_per_process = 1000 # min 25
# (change requires restart)
#shared_preload_libraries = '' # (change requires restart)
# - Cost-Based Vacuum Delay -
#vacuum_cost_delay = 0 # 0-1000 milliseconds
#vacuum_cost_page_hit = 1 # 0-10000 credits
#vacuum_cost_page_miss = 10 # 0-10000 credits
#vacuum_cost_page_dirty = 20 # 0-10000 credits
#vacuum_cost_limit = 200 # 0-10000 credits
# - Background writer -
#bgwriter_delay = 200ms # 10-10000ms between rounds
#bgwriter_lru_percent = 1.0 # 0-100% of LRU buffers scanned/round
#bgwriter_lru_maxpages = 5 # 0-1000 buffers max written/round
#bgwriter_all_percent = 0.333 # 0-100% of all buffers scanned/round
#bgwriter_all_maxpages = 5 # 0-1000 buffers max written/round
#---------------------------------------------------------------------------
# WRITE AHEAD LOG
#---------------------------------------------------------------------------
# - Settings -
#fsync = on # turns forced synchronization on or off
#wal_sync_method = fsync # the default is the first option
# supported by the operating system:
# open_datasync
# fdatasync
# fsync
# fsync_writethrough
# open_sync
#full_page_writes = on # recover from partial page writes
#wal_buffers = 64kB # min 32kB
# (change requires restart)
#commit_delay = 0 # range 0-100000, in microseconds
#commit_siblings = 5 # range 1-1000
# - Checkpoints -
checkpoint_segments = 50 # in logfile segments, min 1, 16MB each
#checkpoint_timeout = 5min # range 30s-1h
#checkpoint_warning = 30s # 0 is off
# - Archiving -
#archive_command = '' # command to use to archive a logfile segment
#archive_timeout = 0 # force a logfile segment switch after this
# many seconds; 0 is off
#---------------------------------------------------------------------------
# QUERY TUNING
#---------------------------------------------------------------------------
# - Planner Method Configuration -
#enable_bitmapscan = on
#enable_hashagg = on
#enable_hashjoin = on
#enable_indexscan = on
#enable_mergejoin = on
#enable_nestloop = on
#enable_seqscan = on
#enable_sort = on
#enable_tidscan = on
# - Planner Cost Constants -
#seq_page_cost = 1.0 # measured on an arbitrary scale
#random_page_cost = 4.0 # same scale as above
#cpu_tuple_cost = 0.01 # same scale as above
#cpu_index_tuple_cost = 0.005 # same scale as above
#cpu_operator_cost = 0.0025 # same scale as above
#effective_cache_size = 128MB
# - Genetic Query Optimizer -
#geqo = on
#geqo_threshold = 12
#geqo_effort = 5 # range 1-10
#geqo_pool_size = 0 # selects default based on effort
#geqo_generations = 0 # selects default based on effort
#geqo_selection_bias = 2.0 # range 1.5-2.0
# - Other Planner Options -
#default_statistics_target = 10 # range 1-1000
#constraint_exclusion = off
#from_collapse_limit = 8
#join_collapse_limit = 8 # 1 disables collapsing of explicit
# JOINs
#---------------------------------------------------------------------------
# ERROR REPORTING AND LOGGING
#---------------------------------------------------------------------------
# - Where to Log -
#log_destination = 'stderr' # Valid values are combinations of
# stderr, syslog and eventlog,
# depending on platform.
# This is used when logging to stderr:
#redirect_stderr = off # Enable capturing of stderr into log
# files
# (change requires restart)
# These are only used if redirect_stderr is on:
#log_directory = 'pg_log' # Directory where log files are written
# Can be absolute or relative to PGDATA
log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log' # Log file name pattern.
# Can include strftime() escapes
log_truncate_on_rotation = off # If on, any existing log file of the same
# name as the new log file will be
# truncated rather than appended to. But
# such truncation only occurs on
# time-driven rotation, not on restarts
# or size-driven rotation. Default is
# off, meaning append to existing files
# in all cases.
#log_rotation_age = 1m # Automatic rotation of logfiles will
# happen after that time. 0 to
# disable.
log_rotation_size = 10MB # Automatic rotation of logfiles will
# happen after that much log
# output. 0 to disable.
# These are relevant when logging to syslog:
#syslog_facility = 'LOCAL0'
#syslog_ident = 'postgres'
# - When to Log -
#client_min_messages = notice # Values, in order of decreasing detail:
# debug5
# debug4
# debug3
# debug2
# debug1
# log
# notice
# warning
# error
#log_min_messages = notice # Values, in order of decreasing detail:
# debug5
# debug4
# debug3
# debug2
# debug1
# info
# notice
# warning
# error
# log
# fatal
# panic
#log_error_verbosity = default # terse, default, or verbose messages
#log_min_error_statement = error # Values in order of increasing severity:
# debug5
# debug4
# debug3
# debug2
# debug1
# info
# notice
# warning
# error
# fatal
# panic (effectively off)
#log_min_duration_statement = -1 # -1 is disabled, 0 logs all statements
# and their durations.
#silent_mode = off # DO NOT USE without syslog or
# redirect_stderr
# (change requires restart)
# - What to Log -
#debug_print_parse = off
#debug_print_rewritten = off
#debug_print_plan = off
#debug_pretty_print = off
#log_connections = off
#log_disconnections = off
#log_duration = off
#log_line_prefix = '' # Special values:
# %u = user name
# %d = database name
# %r = remote host and port
# %h = remote host
# %p = PID
# %t = timestamp (no milliseconds)
# %m = timestamp with milliseconds
# %i = command tag
# %c = session id
# %l = session line number
# %s = session start timestamp
# %x = transaction id
# %q = stop here in non-session
# processes
# %% = '%'
# e.g. '<%u%%%d> '
#log_statement = 'none' # none, ddl, mod, all
#log_hostname = off
#---------------------------------------------------------------------------
# RUNTIME STATISTICS
#---------------------------------------------------------------------------
# - Query/Index Statistics Collector -
#stats_command_string = on
#update_process_title = on
#stats_start_collector = on # needed for block or row stats
# (change requires restart)
#stats_block_level = off
#stats_row_level = off
#stats_reset_on_server_start = off # (change requires restart)
# - Statistics Monitoring -
#log_parser_stats = off
#log_planner_stats = off
#log_executor_stats = off
#log_statement_stats = off
#---------------------------------------------------------------------------
# AUTOVACUUM PARAMETERS
#---------------------------------------------------------------------------
#autovacuum = off # enable autovacuum subprocess?
# 'on' requires stats_start_collector
# and stats_row_level to also be on
#autovacuum_naptime = 1min # time between autovacuum runs
#autovacuum_vacuum_threshold = 500 # min # of tuple updates before
# vacuum
#autovacuum_analyze_threshold = 250 # min # of tuple updates before
# analyze
#autovacuum_vacuum_scale_factor = 0.2 # fraction of rel size before
# vacuum
#autovacuum_analyze_scale_factor = 0.1 # fraction of rel size before
# analyze
#autovacuum_freeze_max_age = 200000000 # maximum XID age before forced vacuum
# (change requires restart)
#autovacuum_vacuum_cost_delay = -1 # default vacuum cost delay for
# autovacuum, -1 means use
# vacuum_cost_delay
#autovacuum_vacuum_cost_limit = -1 # default vacuum cost limit for
# autovacuum, -1 means use
# vacuum_cost_limit
#---------------------------------------------------------------------------
# CLIENT CONNECTION DEFAULTS
#---------------------------------------------------------------------------
# - Statement Behavior -
#search_path = '"$user",public' # schema names
#default_tablespace = '' # a tablespace name, '' uses
# the default
#check_function_bodies = on
#default_transaction_isolation = 'read committed'
#default_transaction_read_only = off
#statement_timeout = 0 # 0 is disabled
#vacuum_freeze_min_age = 100000000
# - Locale and Formatting -
datestyle = 'iso, mdy'
#timezone = unknown # actually, defaults to TZ
# environment setting
#timezone_abbreviations = 'Default' # select the set of available timezone
# abbreviations. Currently, there are
# Default
# Australia
# India
# However you can also create your own
# file in share/timezonesets/.
#extra_float_digits = 0 # min -15, max 2
#client_encoding = sql_ascii # actually, defaults to database
# encoding
# These settings are initialized by initdb -- they might be changed
lc_messages = 'en_US' # locale for system error message
# strings
lc_monetary = 'en_US' # locale for monetary formatting
lc_numeric = 'en_US' # locale for number formatting
lc_time = 'en_US' # locale for time formatting
# - Other Defaults -
#explain_pretty_print = on
#dynamic_library_path = '$libdir'
#local_preload_libraries = ''
#---------------------------------------------------------------------------
# LOCK MANAGEMENT
#---------------------------------------------------------------------------
#deadlock_timeout = 1s
#max_locks_per_transaction = 64 # min 10
# (change requires restart)
# Note: each lock table slot uses ~270 bytes of shared memory, and there are
# max_locks_per_transaction * (max_connections + max_prepared_transactions)
# lock table slots.
#---------------------------------------------------------------------------
# VERSION/PLATFORM COMPATIBILITY
#---------------------------------------------------------------------------
# - Previous Postgres Versions -
#add_missing_from = off
#array_nulls = on
#backslash_quote = safe_encoding # on, off, or safe_encoding
#default_with_oids = off
#escape_string_warning = on
#standard_conforming_strings = off
#regex_flavor = advanced # advanced, extended, or basic
#sql_inheritance = on
# - Other Platforms & Clients -
#transform_null_equals = off
#---------------------------------------------------------------------------
# CUSTOMIZED OPTIONS
#---------------------------------------------------------------------------
#custom_variable_classes = '' # list of custom variable class names
Any suggestions to improve things?
Shawn
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Slow Query
@ 2007-09-03 18:07 Kevin Grittner <Kevin.Grittner@wicourts.gov>
parent: Shawn <postgres@xmtservices.net>
0 siblings, 1 reply; 112+ messages in thread
From: Kevin Grittner @ 2007-09-03 18:07 UTC (permalink / raw)
To: pgsql-performance; Shawn <postgres@xmtservices.net>
>>> On Mon, Sep 3, 2007 at 11:15 AM, in message
<20070903091558.0780b963@boffin.xmtservices.net>, Shawn
<postgres@xmtservices.net> wrote:
> On Sun, 02 Sep 2007 10:49:09 -0500 "Kevin Grittner"
> <Kevin.Grittner@wicourts.gov> wrote:
>
>> >>> On Sat, Sep 1, 2007 at 12:29 PM, in message
>> <20070901102947.0c0a50a3@boffin.xmtservices.net>, Shawn
>> <postgres@xmtservices.net> wrote:
>> > update shawns_data set alias = null;
>> > Even after VACUUM this simple line takes 35 sec to complete.
>>
>> Would any rows already have a null alias when you run this?
>> If so, try adding 'where alias is not null' to the query.
>
> This one initially added about 10sec to the run but I added a HASH
> index on the alias field and its now about 5 sec average runtime, a net
> improvement.
Testing for null on 15,700 rows took five seconds more than the time saved
from not updating some portion of the rows????? I've never seen anything
remotely like that.
Did you ever capture the output of VACUUM VERBOSE against this table (as
Tom requested)?
What happens if you run CLUSTER against this table before running one of
these updates? (Be sure to do that VACUUM VERBOSE first, to see what the
"old" state of the table was, and run it again after.)
What is the row count from the second update of the table in your script?
(An overly loose join there could bloat the table.)
-Kevin
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Slow Query
@ 2007-09-03 18:32 Kevin Grittner <Kevin.Grittner@wicourts.gov>
parent: Shawn <postgres@xmtservices.net>
0 siblings, 0 replies; 112+ messages in thread
From: Kevin Grittner @ 2007-09-03 18:32 UTC (permalink / raw)
To: pgsql-performance; Shawn <postgres@xmtservices.net>
>>> On Mon, Sep 3, 2007 at 11:57 AM, in message
<20070903095749.5045211b@boffin.xmtservices.net>, Shawn
<postgres@xmtservices.net> wrote:
> Also it runs a lot faster by itself
Given the context of the run, there is a possibility that a checkpoint tends
to fall at this point in the script because you're filling your WAL files.
There is a known issue (which people have been working on fixing in 8.3)
which causes the checkpoint to push a lot of pages to the disk and then wait
on physical writes. If the VACUUM ANALYZE doesn't turn up anything useful,
an interesting experiment would be to run your script with these
modifications to your postgresql.conf file:
bgwriter_lru_percent = 20.0 # 0-100% of LRU buffers scanned/round
bgwriter_lru_maxpages = 200 # 0-1000 buffers max written/round
bgwriter_all_percent = 10.0 # 0-100% of all buffers scanned/round
bgwriter_all_maxpages = 600 # 0-1000 buffers max written/round
Don't leave these in effect permanently without close attention to the
overall impact. These settings have worked well for us, but are likely
not to work well for everyone.
-Kevin
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Slow Query
@ 2007-09-03 23:53 Shawn <postgres@xmtservices.net>
parent: Kevin Grittner <Kevin.Grittner@wicourts.gov>
0 siblings, 2 replies; 112+ messages in thread
From: Shawn @ 2007-09-03 23:53 UTC (permalink / raw)
To: pgsql-performance
On Mon, 03 Sep 2007 13:07:41 -0500
"Kevin Grittner" <Kevin.Grittner@wicourts.gov> wrote:
> >>> On Mon, Sep 3, 2007 at 11:15 AM, in message
> <20070903091558.0780b963@boffin.xmtservices.net>, Shawn
> <postgres@xmtservices.net> wrote:
> > On Sun, 02 Sep 2007 10:49:09 -0500 "Kevin Grittner"
> > <Kevin.Grittner@wicourts.gov> wrote:
> >
> >> >>> On Sat, Sep 1, 2007 at 12:29 PM, in message
> >> <20070901102947.0c0a50a3@boffin.xmtservices.net>, Shawn
> >> <postgres@xmtservices.net> wrote:
> >> > update shawns_data set alias = null;
> >> > Even after VACUUM this simple line takes 35 sec to complete.
> >>
> >> Would any rows already have a null alias when you run this?
> >> If so, try adding 'where alias is not null' to the query.
> >
> > This one initially added about 10sec to the run but I added a HASH
> > index on the alias field and its now about 5 sec average runtime, a
> > net improvement.
>
> Testing for null on 15,700 rows took five seconds more than the time
> saved from not updating some portion of the rows????? I've never
> seen anything remotely like that.
>
> Did you ever capture the output of VACUUM VERBOSE against this table
> (as Tom requested)?
>
> What happens if you run CLUSTER against this table before running one
> of these updates? (Be sure to do that VACUUM VERBOSE first, to see
> what the "old" state of the table was, and run it again after.)
>
> What is the row count from the second update of the table in your
> script? (An overly loose join there could bloat the table.)
here is the vacuum results:
vacuum verbose analyze shawns_data;
INFO: vacuuming "public.shawns_data"
INFO: scanned index "shawns_data_pkey" to remove 21444 row versions
DETAIL: CPU 0.24s/0.12u sec elapsed 8.35 sec.
INFO: scanned index "sd_l" to remove 21444 row versions
DETAIL: CPU 0.32s/0.16u sec elapsed 6.11 sec.
INFO: scanned index "sd_b" to remove 21444 row versions
DETAIL: CPU 0.34s/0.13u sec elapsed 10.10 sec.
INFO: scanned index "sd_s" to remove 21444 row versions
DETAIL: CPU 0.36s/0.13u sec elapsed 7.16 sec.
INFO: scanned index "sd_e" to remove 21444 row versions
DETAIL: CPU 0.40s/0.17u sec elapsed 6.71 sec.
INFO: scanned index "sd_alias_hash" to remove 21444 row versions
DETAIL: CPU 0.00s/0.01u sec elapsed 0.01 sec.
INFO: "shawns_data": removed 21444 row versions in 513 pages
DETAIL: CPU 0.00s/0.00u sec elapsed 0.00 sec.
INFO: index "shawns_data_pkey" now contains 15445 row versions in
35230 pages DETAIL: 21444 index row versions were removed.
19255 index pages have been deleted, 19255 are currently reusable.
CPU 0.00s/0.00u sec elapsed 0.00 sec.
INFO: index "sd_l" now contains 15445 row versions in 32569 pages
DETAIL: 21444 index row versions were removed.
18059 index pages have been deleted, 18059 are currently reusable.
CPU 0.00s/0.00u sec elapsed 0.00 sec.
INFO: index "sd_b" now contains 15445 row versions in 34119 pages
DETAIL: 21444 index row versions were removed.
30276 index pages have been deleted, 30219 are currently reusable.
CPU 0.00s/0.00u sec elapsed 0.00 sec.
INFO: index "sd_s" now contains 15445 row versions in 35700 pages
DETAIL: 21444 index row versions were removed.
31284 index pages have been deleted, 31233 are currently reusable.
CPU 0.00s/0.00u sec elapsed 0.00 sec.
INFO: index "sd_e" now contains 15445 row versions in 42333 pages
DETAIL: 21444 index row versions were removed.
28828 index pages have been deleted, 28820 are currently reusable.
CPU 0.00s/0.00u sec elapsed 0.00 sec.
INFO: index "sd_alias_hash" now contains 10722 row versions in 298
pages DETAIL: 10722 index row versions were removed.
0 index pages have been deleted, 0 are currently reusable.
CPU 0.00s/0.00u sec elapsed 0.00 sec.
INFO: "shawns_data": found 21444 removable, 15445 nonremovable row
versions in 770 pages DETAIL: 0 dead row versions cannot be removed
yet. There were 5825 unused item pointers.
543 pages contain useful free space.
0 pages are entirely empty.
CPU 1.68s/0.77u sec elapsed 38.47 sec.
INFO: analyzing "public.shawns_data"
INFO: "shawns_data": scanned 770 of 770 pages, containing 15445 live
rows and 0 dead rows; 3000 rows in sample, 15445 estimated total rows
VACUUM
Shawn
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Slow Query
@ 2007-09-04 00:49 Tom Lane <tgl@sss.pgh.pa.us>
parent: Shawn <postgres@xmtservices.net>
1 sibling, 1 reply; 112+ messages in thread
From: Tom Lane @ 2007-09-04 00:49 UTC (permalink / raw)
To: Shawn <postgres@xmtservices.net>; +Cc: pgsql-performance
Shawn <postgres@xmtservices.net> writes:
> "Kevin Grittner" <Kevin.Grittner@wicourts.gov> wrote:
>> Did you ever capture the output of VACUUM VERBOSE against this table
> vacuum verbose analyze shawns_data;
> ...
> INFO: index "shawns_data_pkey" now contains 15445 row versions in
> 35230 pages
[ and comparably bloated sizes for other indexes ]
Ouch! The table itself doesn't look nearly as bad:
> INFO: "shawns_data": found 21444 removable, 15445 nonremovable row
> versions in 770 pages
but you've got a spectacularly bad case of index bloat. An index 50
times bigger than its table is Not Good. I think you'd find that
"REINDEX TABLE shawns_data" does wonders for the situation.
The next question is how it got to be that way... what is your
vacuuming policy for this database? Maybe you need to raise
max_fsm_pages?
regards, tom lane
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Slow Query
@ 2007-09-04 02:40 Kevin Grittner <Kevin.Grittner@wicourts.gov>
parent: Shawn <postgres@xmtservices.net>
1 sibling, 0 replies; 112+ messages in thread
From: Kevin Grittner @ 2007-09-04 02:40 UTC (permalink / raw)
To: pgsql-performance; Shawn <postgres@xmtservices.net>
>>> On Mon, Sep 3, 2007 at 6:53 PM, in message
<20070903165334.0da769c1@boffin.xmtservices.net>, Shawn
<postgres@xmtservices.net> wrote:
> vacuum verbose analyze shawns_data;
> INFO: vacuuming "public.shawns_data"
> INFO: scanned index "shawns_data_pkey" to remove 21444 row versions
> DETAIL: CPU 0.24s/0.12u sec elapsed 8.35 sec.
> INFO: scanned index "sd_l" to remove 21444 row versions
> DETAIL: CPU 0.32s/0.16u sec elapsed 6.11 sec.
> INFO: scanned index "sd_b" to remove 21444 row versions
> DETAIL: CPU 0.34s/0.13u sec elapsed 10.10 sec.
> INFO: scanned index "sd_s" to remove 21444 row versions
> DETAIL: CPU 0.36s/0.13u sec elapsed 7.16 sec.
> INFO: scanned index "sd_e" to remove 21444 row versions
> DETAIL: CPU 0.40s/0.17u sec elapsed 6.71 sec.
> INFO: scanned index "sd_alias_hash" to remove 21444 row versions
> DETAIL: CPU 0.00s/0.01u sec elapsed 0.01 sec.
> INFO: "shawns_data": removed 21444 row versions in 513 pages
> DETAIL: CPU 0.00s/0.00u sec elapsed 0.00 sec.
> INFO: index "shawns_data_pkey" now contains 15445 row versions in
> 35230 pages DETAIL: 21444 index row versions were removed.
> 19255 index pages have been deleted, 19255 are currently reusable.
> CPU 0.00s/0.00u sec elapsed 0.00 sec.
> INFO: index "sd_l" now contains 15445 row versions in 32569 pages
> DETAIL: 21444 index row versions were removed.
> 18059 index pages have been deleted, 18059 are currently reusable.
> CPU 0.00s/0.00u sec elapsed 0.00 sec.
> INFO: index "sd_b" now contains 15445 row versions in 34119 pages
> DETAIL: 21444 index row versions were removed.
> 30276 index pages have been deleted, 30219 are currently reusable.
> CPU 0.00s/0.00u sec elapsed 0.00 sec.
> INFO: index "sd_s" now contains 15445 row versions in 35700 pages
> DETAIL: 21444 index row versions were removed.
> 31284 index pages have been deleted, 31233 are currently reusable.
> CPU 0.00s/0.00u sec elapsed 0.00 sec.
> INFO: index "sd_e" now contains 15445 row versions in 42333 pages
> DETAIL: 21444 index row versions were removed.
> 28828 index pages have been deleted, 28820 are currently reusable.
> CPU 0.00s/0.00u sec elapsed 0.00 sec.
> INFO: index "sd_alias_hash" now contains 10722 row versions in 298
> pages DETAIL: 10722 index row versions were removed.
> 0 index pages have been deleted, 0 are currently reusable.
> CPU 0.00s/0.00u sec elapsed 0.00 sec.
> INFO: "shawns_data": found 21444 removable, 15445 nonremovable row
> versions in 770 pages DETAIL: 0 dead row versions cannot be removed
> yet. There were 5825 unused item pointers.
> 543 pages contain useful free space.
> 0 pages are entirely empty.
> CPU 1.68s/0.77u sec elapsed 38.47 sec.
Those indexes are killing you. Hopefully you realize that each of those
indexes will have a new entry inserted whenever you update a row. If your
indexes are that expensive to maintain, you want to go out of your way
update rows only when something actually changes, which is not the case
for your second update statement yet.
I don't recall seeing the table definition yet. Could we see that, with
the indexes? Also, have you tried that CLUSTER yet? Syntax:
CLUSTER shawns_data_pkey ON shawns_data;
ANALYZE shawns_data;
(or VACUUM ANALYZE)
This will clean up index bloat.
-Kevin
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Slow Query
@ 2007-09-10 01:31 Shawn <postgres@xmtservices.net>
parent: Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 0 replies; 112+ messages in thread
From: Shawn @ 2007-09-10 01:31 UTC (permalink / raw)
To: pgsql-performance
On Mon, 03 Sep 2007 21:10:06 -0400
Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Shawn <postgres@xmtservices.net> writes:
> > You weren't kidding, that really made a difference. its .5 sec now
> > on the run. I think the vacuuming not running for a few weeks is
> > what got me. I caught an edit that had stopped the vacuuming
> > routine a while back. That must have been what caused the index
> > bloat.
>
> Hmm, but no vacuuming at all would have allowed the table itself to
> bloat too. Did you do a VACUUM FULL to recover after you noticed
> the size problem? The trouble with VACUUM FULL is it compresses the
> table but doesn't do a dang thing for indexes ...
>
> regards, tom lane
>
Thanks again to the list. I ran the reindex command that Tom suggested
and things seem to be holding. Even the vacuum command is running
faster. The whole script runs in about 10 seconds, far more acceptable.
Thanks Guys!
Shawn
^ permalink raw reply [nested|flat] 112+ messages in thread
* Slow Query
@ 2007-12-11 15:02 Pallav Kalva <pkalva@livedatagroup.com>
0 siblings, 0 replies; 112+ messages in thread
From: Pallav Kalva @ 2007-12-11 15:02 UTC (permalink / raw)
To: pgsql-performance
Hi,
This below query is taking more than 3 minutes to run, as you can see
from the explain plan it is pretty much using all the indexes still it
is slow, nested loops are taking too long. Is there anyway I can improve
this query performance ?
I am using postgres8.2.4. Here are the number of records in each table.
helix_fdc=# select relname,relpages,reltuples from pg_class where
relname in
('activity','listingactivity','activitytype','listing','address');
relname | relpages | reltuples
-----------------+----------+-------------
listing | 122215 | 8.56868e+06
listingactivity | 51225 | 8.67308e+06
address | 244904 | 1.5182e+07
activity | 733896 | 6.74342e+07
activitytype | 2 | 120
helix_fdc=# explain analyze
helix_fdc-# select count(listingact0_.listingactivityid) as col_0_0_,
date_trunc('day', activity3_.createdate) as col_1_0_,
helix_fdc-# activityty1_.activitytypeid as col_2_0_,
zipcode2_.zipcodeId as col_3_0_
helix_fdc-# from listing.listingactivity listingact0_, common.activity
activity3_, common.activitytype activityty1_,
helix_fdc-# postal.zipcode zipcode2_, common.activitytype
activityty5_, listing.listing listing7_,
helix_fdc-# listing.address listingadd8_
helix_fdc-# where listingact0_.fkactivityid=activity3_.activityId
helix_fdc-# and activity3_.fkactivitytypeid=activityty5_.activitytypeid
helix_fdc-# and listingact0_.fklistingid=listing7_.listingid
helix_fdc-# and listing7_.fkbestaddressid=listingadd8_.addressid
helix_fdc-# and (activityty5_.name in ( 'LISTING_ELEMENT_DETAIL',
'VIRTUALCARD_DISPLAY'))
helix_fdc-# and activity3_.fkactivitytypeid=activityty1_.activitytypeid
helix_fdc-# and listingadd8_.fkzipcodeid=zipcode2_.zipcodeId
helix_fdc-# and (listingadd8_.fkzipcodeid is not null)
helix_fdc-# and activity3_.createdate>='2007-12-11 00:00:00'
helix_fdc-# group by date_trunc('day', activity3_.createdate) ,
activityty1_.activitytypeid , zipcode2_.zipcodeId;
QUERY PLAN
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
HashAggregate (cost=3587.71..3588.31 rows=40 width=20) (actual
time=214022.231..214025.829 rows=925 loops=1)
-> Nested Loop (cost=3.52..3587.31 rows=40 width=20) (actual
time=464.743..213996.150 rows=3571 loops=1)
-> Nested Loop (cost=3.52..3574.01 rows=40 width=24) (actual
time=461.514..213891.251 rows=3571 loops=1)
-> Nested Loop (cost=3.52..3469.18 rows=41 width=24)
(actual time=421.683..208158.769 rows=3571 loops=1)
-> Nested Loop (cost=3.52..3299.05 rows=41
width=24) (actual time=321.155..91460.769 rows=3586 loops=1)
-> Nested Loop (cost=3.52..3147.50 rows=41
width=24) (actual time=188.756..821.893 rows=3586 loops=1)
-> Hash Join (cost=3.52..880.59
rows=321 width=20) (actual time=103.689..325.236 rows=4082 loops=1)
Hash Cond:
(activity3_.fkactivitytypeid = activityty5_.activitytypeid)
-> Index Scan using
idx_activity_createdate on activity activity3_ (cost=0.00..801.68
rows=19247 width=16) (actual time=103.495..244.987 rows=16918 loops=1)
Index Cond: (createdate >=
'2007-12-11 00:00:00'::timestamp without time zone)
-> Hash (cost=3.50..3.50 rows=2
width=4) (actual time=0.148..0.148 rows=2 loops=1)
-> Seq Scan on
activitytype activityty5_ (cost=0.00..3.50 rows=2 width=4) (actual
time=0.062..0.128 rows=2 loops=1)
Filter: (name = ANY
('{LISTING_ELEMENT_DETAIL,VIRTUALCARD_DISPLAY}'::text[]))
-> Index Scan using
idx_listingactivity_fkactivityid on listingactivity listingact0_
(cost=0.00..7.05 rows=1 width=12) (actual time=0.097..0.108 rows=1
loops=4082)
Index Cond:
(listingact0_.fkactivityid = activity3_.activityid)
-> Index Scan using pk_listing_listingid on
listing listing7_ (cost=0.00..3.68 rows=1 width=8) (actual
time=25.216..25.260 rows=1 loops=3586)
Index Cond: (listingact0_.fklistingid =
listing7_.listingid)
-> Index Scan using pk_address_addressid on
address listingadd8_ (cost=0.00..4.14 rows=1 width=8) (actual
time=32.508..32.527 rows=1 loops=3586)
Index Cond: (listing7_.fkbestaddressid =
listingadd8_.addressid)
Filter: (fkzipcodeid IS NOT NULL)
-> Index Scan using pk_zipcode_zipcodeid on zipcode
zipcode2_ (cost=0.00..2.54 rows=1 width=4) (actual time=1.586..1.590
rows=1 loops=3571)
Index Cond: (listingadd8_.fkzipcodeid =
zipcode2_.zipcodeid)
-> Index Scan using pk_activitytype_activitytypeid on
activitytype activityty1_ (cost=0.00..0.32 rows=1 width=4) (actual
time=0.007..0.011 rows=1 loops=3571)
Index Cond: (activity3_.fkactivitytypeid =
activityty1_.activitytypeid)
Total runtime: 214029.185 ms
(25 rows)
^ permalink raw reply [nested|flat] 112+ messages in thread
* slow query
@ 2010-06-04 01:45 Anj Adu <fotographs@gmail.com>
0 siblings, 3 replies; 112+ messages in thread
From: Anj Adu @ 2010-06-04 01:45 UTC (permalink / raw)
To: pgsql-performance
I am reposting as my original query was mangled
The link to the explain plan is here as it does not paste well into
the email body.
http://explain.depesz.com/s/kHa
The machine is a 2 cpu quad core 5430 with 32G RAM and 6x450G 15K
single raid-10 array
1G work_mem
default_statistics_target=1000
random_page_cost=1
I am curious why the hash join takes so long. The main table
dev4_act_dy_fact_2010_05_t has 25 million rows. The table is
partitioned into 3 parts per month. Remaining tables are very small (
< 1000 rows)
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: slow query
@ 2010-06-04 08:13 tv@fuzzy.cz
parent: Anj Adu <fotographs@gmail.com>
2 siblings, 1 reply; 112+ messages in thread
From: tv@fuzzy.cz @ 2010-06-04 08:13 UTC (permalink / raw)
To: Anj Adu <fotographs@gmail.com>; +Cc: pgsql-performance
> I am reposting as my original query was mangled
>
> The link to the explain plan is here as it does not paste well into
> the email body.
>
> http://explain.depesz.com/s/kHa
>
>
> The machine is a 2 cpu quad core 5430 with 32G RAM and 6x450G 15K
> single raid-10 array
>
> 1G work_mem
> default_statistics_target=1000
> random_page_cost=1
Are you sure it's wise to set the work_mem to 1G? Do you really need it?
Don't forget this is not a 'total' or 'per query' - each query may
allocate multiple work areas (and occupy multiple GB). But I guess this
does not cause the original problem.
The last row 'random_page_cost=1' - this basically says that reading data
by random is just as cheap as reading data sequentially. Which may result
in poor performance due to bad plans. Why have you set this value?
Sure, there are rare cases where 'random_page_cost=1' is OK.
>
> I am curious why the hash join takes so long. The main table
> dev4_act_dy_fact_2010_05_t has 25 million rows. The table is
> partitioned into 3 parts per month. Remaining tables are very small (
> < 1000 rows)
Well, the real cause that makes your query slow is the 'index scan' part.
Index Scan using dev4_act_dy_fact_2010_05_t3_thedate on
dev4_act_dy_fact_2010_05_t3 a (cost=0.00..94041.89 rows=204276 width=60)
(actual time=164533.725..164533.725 rows=0 loops=1)
The first thing to note here is the difference in expected and actual
number of rows - the planner expects 204276 but gets 0 rows. How large is
this partition?
Try to analyze it, set the random_page_cost to something reasonable (e.g.
4) and try to run the query again.
Tomas
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: slow query
@ 2010-06-04 09:00 Matthew Wakeling <matthew@flymine.org>
parent: Anj Adu <fotographs@gmail.com>
2 siblings, 1 reply; 112+ messages in thread
From: Matthew Wakeling @ 2010-06-04 09:00 UTC (permalink / raw)
To: Anj Adu <fotographs@gmail.com>; +Cc: pgsql-performance
On Thu, 3 Jun 2010, Anj Adu wrote:
> http://explain.depesz.com/s/kHa
I'm interested in why the two partitions dev4_act_dy_fact and
dev4_act_dy_fact_2010_05_t3 are treated so differently. I'm guessing that
the former is the parent and the latter the child table?
When accessing the parent table, Postgres is able to use a bitmap AND
index scan, because it has the two indexes dev4_act_dy_dm_nd_indx and
dev4_act_dy_dm_cd_indx. Do the child tables have a similar index setup?
Incidentally, you could get even better than a bitmap AND index scan by
creating an index on (node_id, thedate) on each table.
> random_page_cost=1
I agree with Tomas that this is rarely a useful setting.
Matthew
--
You can configure Windows, but don't ask me how. -- Bill Gates
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: slow query
@ 2010-06-04 17:25 Anj Adu <fotographs@gmail.com>
parent: Matthew Wakeling <matthew@flymine.org>
0 siblings, 0 replies; 112+ messages in thread
From: Anj Adu @ 2010-06-04 17:25 UTC (permalink / raw)
To: Matthew Wakeling <matthew@flymine.org>; +Cc: pgsql-performance
I'm interested in why the two partitions dev4_act_dy_fact and
> dev4_act_dy_fact_2010_05_t3 are treated so differently. I'm guessing that
> the former is the parent and the latter the child table?
Yes..you are correct.
>
> When accessing the parent table, Postgres is able to use a bitmap AND index
> scan, because it has the two indexes dev4_act_dy_dm_nd_indx and
> dev4_act_dy_dm_cd_indx. Do the child tables have a similar index setup?
Yes..the child table have indexes on those fields as well
>
> Incidentally, you could get even better than a bitmap AND index scan by
> creating an index on (node_id, thedate) on each table.
Will this perform better than separate indexes ?
>
>> random_page_cost=1
>
> I agree with Tomas that this is rarely a useful setting.
>
> Matthew
>
> --
> You can configure Windows, but don't ask me how. -- Bill Gates
>
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: slow query
@ 2010-06-04 17:41 Anj Adu <fotographs@gmail.com>
parent: tv@fuzzy.cz
0 siblings, 1 reply; 112+ messages in thread
From: Anj Adu @ 2010-06-04 17:41 UTC (permalink / raw)
To: tv@fuzzy.cz; +Cc: pgsql-performance
2010/6/4 <tv@fuzzy.cz>:
>> I am reposting as my original query was mangled
>>
>> The link to the explain plan is here as it does not paste well into
>> the email body.
>>
>> http://explain.depesz.com/s/kHa
>>
>>
>> The machine is a 2 cpu quad core 5430 with 32G RAM and 6x450G 15K
>> single raid-10 array
>>
>> 1G work_mem
>> default_statistics_target=1000
>> random_page_cost=1
>
> Are you sure it's wise to set the work_mem to 1G? Do you really need it?
> Don't forget this is not a 'total' or 'per query' - each query may
> allocate multiple work areas (and occupy multiple GB). But I guess this
> does not cause the original problem.
>
> The last row 'random_page_cost=1' - this basically says that reading data
> by random is just as cheap as reading data sequentially. Which may result
> in poor performance due to bad plans. Why have you set this value?
>
> Sure, there are rare cases where 'random_page_cost=1' is OK.
The default for 8.4 is 2
I tried with 2 and 1..but the results are not very different. I
understand that for fast disks (which we have with a decent Raid 10
setup)..the random_page_cost can be lowered as needed..but I guess it
did not make a difference here.
>
>>
>> I am curious why the hash join takes so long. The main table
>> dev4_act_dy_fact_2010_05_t has 25 million rows. The table is
>> partitioned into 3 parts per month. Remaining tables are very small (
>> < 1000 rows)
>
> Well, the real cause that makes your query slow is the 'index scan' part.
>
> Index Scan using dev4_act_dy_fact_2010_05_t3_thedate on
> dev4_act_dy_fact_2010_05_t3 a (cost=0.00..94041.89 rows=204276 width=60)
> (actual time=164533.725..164533.725 rows=0 loops=1)
>
> The first thing to note here is the difference in expected and actual
> number of rows - the planner expects 204276 but gets 0 rows. How large is
> this partition?
The partition has 25 million rows with indexes on theDate, node_id..
I altered the random_page_cost to 4 (1 more than the default)..still
slow. These tables are analyzed every day
I have an index on each field used in the where criteria,
>
> Try to analyze it, set the random_page_cost to something reasonable (e.g.
> 4) and try to run the query again.
>
> Tomas
>
>
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: slow query
@ 2010-06-04 18:01 Anj Adu <fotographs@gmail.com>
parent: Anj Adu <fotographs@gmail.com>
0 siblings, 1 reply; 112+ messages in thread
From: Anj Adu @ 2010-06-04 18:01 UTC (permalink / raw)
To: tv@fuzzy.cz; +Cc: pgsql-performance
Does the difference in expected and actual rows as seen by the planner
a big factor? Even after an analyze...the results are similar. (there
is a big diff between expected and actual)
Partition has 25 million rows
On Fri, Jun 4, 2010 at 10:41 AM, Anj Adu <fotographs@gmail.com> wrote:
> 2010/6/4 <tv@fuzzy.cz>:
>>> I am reposting as my original query was mangled
>>>
>>> The link to the explain plan is here as it does not paste well into
>>> the email body.
>>>
>>> http://explain.depesz.com/s/kHa
>>>
>>>
>>> The machine is a 2 cpu quad core 5430 with 32G RAM and 6x450G 15K
>>> single raid-10 array
>>>
>>> 1G work_mem
>>> default_statistics_target=1000
>>> random_page_cost=1
>>
>> Are you sure it's wise to set the work_mem to 1G? Do you really need it?
>> Don't forget this is not a 'total' or 'per query' - each query may
>> allocate multiple work areas (and occupy multiple GB). But I guess this
>> does not cause the original problem.
>>
>> The last row 'random_page_cost=1' - this basically says that reading data
>> by random is just as cheap as reading data sequentially. Which may result
>> in poor performance due to bad plans. Why have you set this value?
>>
>> Sure, there are rare cases where 'random_page_cost=1' is OK.
>
> The default for 8.4 is 2
> I tried with 2 and 1..but the results are not very different. I
> understand that for fast disks (which we have with a decent Raid 10
> setup)..the random_page_cost can be lowered as needed..but I guess it
> did not make a difference here.
>
>
>>
>>>
>>> I am curious why the hash join takes so long. The main table
>>> dev4_act_dy_fact_2010_05_t has 25 million rows. The table is
>>> partitioned into 3 parts per month. Remaining tables are very small (
>>> < 1000 rows)
>>
>> Well, the real cause that makes your query slow is the 'index scan' part.
>>
>> Index Scan using dev4_act_dy_fact_2010_05_t3_thedate on
>> dev4_act_dy_fact_2010_05_t3 a (cost=0.00..94041.89 rows=204276 width=60)
>> (actual time=164533.725..164533.725 rows=0 loops=1)
>>
>> The first thing to note here is the difference in expected and actual
>> number of rows - the planner expects 204276 but gets 0 rows. How large is
>> this partition?
>
> The partition has 25 million rows with indexes on theDate, node_id..
> I altered the random_page_cost to 4 (1 more than the default)..still
> slow. These tables are analyzed every day
> I have an index on each field used in the where criteria,
>>
>> Try to analyze it, set the random_page_cost to something reasonable (e.g.
>> 4) and try to run the query again.
>>
>> Tomas
>>
>>
>
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: slow query
@ 2010-06-04 18:21 Anj Adu <fotographs@gmail.com>
parent: Anj Adu <fotographs@gmail.com>
0 siblings, 1 reply; 112+ messages in thread
From: Anj Adu @ 2010-06-04 18:21 UTC (permalink / raw)
To: tv@fuzzy.cz; +Cc: pgsql-performance
The behaviour is different in postgres 8.1.9 (much faster) (the table
has 9 million rows instead of 25 million..but the query comes back
very fast (8 seconds)..
Wonder if this is very specific to 8.4.0
On Fri, Jun 4, 2010 at 11:01 AM, Anj Adu <fotographs@gmail.com> wrote:
> Does the difference in expected and actual rows as seen by the planner
> a big factor? Even after an analyze...the results are similar. (there
> is a big diff between expected and actual)
> Partition has 25 million rows
>
> On Fri, Jun 4, 2010 at 10:41 AM, Anj Adu <fotographs@gmail.com> wrote:
>> 2010/6/4 <tv@fuzzy.cz>:
>>>> I am reposting as my original query was mangled
>>>>
>>>> The link to the explain plan is here as it does not paste well into
>>>> the email body.
>>>>
>>>> http://explain.depesz.com/s/kHa
>>>>
>>>>
>>>> The machine is a 2 cpu quad core 5430 with 32G RAM and 6x450G 15K
>>>> single raid-10 array
>>>>
>>>> 1G work_mem
>>>> default_statistics_target=1000
>>>> random_page_cost=1
>>>
>>> Are you sure it's wise to set the work_mem to 1G? Do you really need it?
>>> Don't forget this is not a 'total' or 'per query' - each query may
>>> allocate multiple work areas (and occupy multiple GB). But I guess this
>>> does not cause the original problem.
>>>
>>> The last row 'random_page_cost=1' - this basically says that reading data
>>> by random is just as cheap as reading data sequentially. Which may result
>>> in poor performance due to bad plans. Why have you set this value?
>>>
>>> Sure, there are rare cases where 'random_page_cost=1' is OK.
>>
>> The default for 8.4 is 2
>> I tried with 2 and 1..but the results are not very different. I
>> understand that for fast disks (which we have with a decent Raid 10
>> setup)..the random_page_cost can be lowered as needed..but I guess it
>> did not make a difference here.
>>
>>
>>>
>>>>
>>>> I am curious why the hash join takes so long. The main table
>>>> dev4_act_dy_fact_2010_05_t has 25 million rows. The table is
>>>> partitioned into 3 parts per month. Remaining tables are very small (
>>>> < 1000 rows)
>>>
>>> Well, the real cause that makes your query slow is the 'index scan' part.
>>>
>>> Index Scan using dev4_act_dy_fact_2010_05_t3_thedate on
>>> dev4_act_dy_fact_2010_05_t3 a (cost=0.00..94041.89 rows=204276 width=60)
>>> (actual time=164533.725..164533.725 rows=0 loops=1)
>>>
>>> The first thing to note here is the difference in expected and actual
>>> number of rows - the planner expects 204276 but gets 0 rows. How large is
>>> this partition?
>>
>> The partition has 25 million rows with indexes on theDate, node_id..
>> I altered the random_page_cost to 4 (1 more than the default)..still
>> slow. These tables are analyzed every day
>> I have an index on each field used in the where criteria,
>>>
>>> Try to analyze it, set the random_page_cost to something reasonable (e.g.
>>> 4) and try to run the query again.
>>>
>>> Tomas
>>>
>>>
>>
>
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: slow query
@ 2010-06-05 09:02 hubert depesz lubaczewski <depesz@depesz.com>
parent: Anj Adu <fotographs@gmail.com>
2 siblings, 0 replies; 112+ messages in thread
From: hubert depesz lubaczewski @ 2010-06-05 09:02 UTC (permalink / raw)
To: Anj Adu <fotographs@gmail.com>; +Cc: pgsql-performance
On Thu, Jun 03, 2010 at 06:45:30PM -0700, Anj Adu wrote:
> http://explain.depesz.com/s/kHa
can you please show us \d dev4_act_dy_fact_2010_05_t3 ?
depesz
--
Linkedin: http://www.linkedin.com/in/depesz / blog: http://www.depesz.com/
jid/gtalk: depesz@depesz.com / aim:depeszhdl / skype:depesz_hdl / gg:6749007
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: slow query
@ 2010-06-05 09:38 Scott Marlowe <scott.marlowe@gmail.com>
parent: Anj Adu <fotographs@gmail.com>
0 siblings, 1 reply; 112+ messages in thread
From: Scott Marlowe @ 2010-06-05 09:38 UTC (permalink / raw)
To: Anj Adu <fotographs@gmail.com>; +Cc: tv@fuzzy.cz; pgsql-performance
On Fri, Jun 4, 2010 at 12:21 PM, Anj Adu <fotographs@gmail.com> wrote:
> The behaviour is different in postgres 8.1.9 (much faster) (the table
> has 9 million rows instead of 25 million..but the query comes back
> very fast (8 seconds)..
>
> Wonder if this is very specific to 8.4.0
You should really be running 8.4.4.
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: slow query
@ 2010-06-05 14:02 Anj Adu <fotographs@gmail.com>
parent: Scott Marlowe <scott.marlowe@gmail.com>
0 siblings, 1 reply; 112+ messages in thread
From: Anj Adu @ 2010-06-05 14:02 UTC (permalink / raw)
To: Scott Marlowe <scott.marlowe@gmail.com>; +Cc: tv@fuzzy.cz; pgsql-performance
Thanks..I'll try this. Should I also rebuild the contrib modules..or
just the core postgres database?
On Sat, Jun 5, 2010 at 2:38 AM, Scott Marlowe <scott.marlowe@gmail.com> wrote:
> On Fri, Jun 4, 2010 at 12:21 PM, Anj Adu <fotographs@gmail.com> wrote:
>> The behaviour is different in postgres 8.1.9 (much faster) (the table
>> has 9 million rows instead of 25 million..but the query comes back
>> very fast (8 seconds)..
>>
>> Wonder if this is very specific to 8.4.0
>
> You should really be running 8.4.4.
>
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: slow query
@ 2010-06-05 21:25 Scott Marlowe <scott.marlowe@gmail.com>
parent: Anj Adu <fotographs@gmail.com>
0 siblings, 0 replies; 112+ messages in thread
From: Scott Marlowe @ 2010-06-05 21:25 UTC (permalink / raw)
To: Anj Adu <fotographs@gmail.com>; +Cc: tv@fuzzy.cz; pgsql-performance
On Sat, Jun 5, 2010 at 8:02 AM, Anj Adu <fotographs@gmail.com> wrote:
> Thanks..I'll try this. Should I also rebuild the contrib modules..or
> just the core postgres database?
That's really up to you. If you use a contrib module in particular,
I'd definitely rebuild that one. It's pretty easy anyway.
^ permalink raw reply [nested|flat] 112+ messages in thread
* Slow Query
@ 2010-08-27 00:03 Ozer, Pam <pozer@automotive.com>
0 siblings, 4 replies; 112+ messages in thread
From: Ozer, Pam @ 2010-08-27 00:03 UTC (permalink / raw)
To: pgsql-performance
I am new to Postgres and I am trying to understand the Explain Analyze
so I can tune the following query. I run the same query using mysql and
it takes less than 50ms. I run it on postgres and it takes 10 seconds.
I feel like I am missing something very obvious. (VehicleUsed is a big
table over 750,000records) and datasetgroupyearmakemodel has 150000
records.
It looks like the cost is highest in the Hash Join on Postalcode. Am
I reading this correctly.? I do have indexes on the lower(postalcode)
in both tables. Why wouldn't be using the index? Thanks in advance for
any help.
Here is my query:
select distinct VehicleMake.VehicleMake
from VehicleUsed
inner join PostalCodeRegionCountyCity on ( lower (
VehicleUsed.PostalCode ) = lower ( PostalCodeRegionCountyCity.PostalCode
) )
INNER JOIN DATASETGROUPYEARMAKEMODEL ON ( VEHICLEUSED.VEHICLEYEAR =
DATASETGROUPYEARMAKEMODEL.VEHICLEYEAR )
AND ( VEHICLEUSED.VEHICLEMAKEID =
DATASETGROUPYEARMAKEMODEL.VEHICLEMAKEID )
AND ( VEHICLEUSED.VEHICLEMODELID =
DATASETGROUPYEARMAKEMODEL.VEHICLEMODELID )
inner join VehicleMake on ( VehicleUsed.VehicleMakeId =
VehicleMake.VehicleMakeId )
where
( DatasetGroupYearMakeModel.DatasetGroupId = 3 ) and
( VehicleUsed.DatasetId <> 113 )
and ( VehicleUsed.ProductGroupId <> 13 )
and ( PostalCodeRegionCountyCity.RegionId = 36 )
order by VehicleMake.VehicleMake
limit 500000
Here is the explain analyze
"Limit (cost=38292.53..38293.19 rows=261 width=8) (actual
time=10675.857..10675.892 rows=42 loops=1)"
" -> Sort (cost=38292.53..38293.19 rows=261 width=8) (actual
time=10675.855..10675.868 rows=42 loops=1)"
" Sort Key: vehiclemake.vehiclemake"
" Sort Method: quicksort Memory: 18kB"
" -> HashAggregate (cost=38279.45..38282.06 rows=261 width=8)
(actual time=10675.710..10675.728 rows=42 loops=1)"
" -> Hash Join (cost=436.31..38270.51 rows=3576 width=8)
(actual time=4.471..10658.291 rows=10425 loops=1)"
" Hash Cond: (vehicleused.vehiclemakeid =
vehiclemake.vehiclemakeid)"
" -> Hash Join (cost=428.43..38213.47 rows=3576
width=4) (actual time=4.152..10639.742 rows=10425 loops=1)"
" Hash Cond:
(lower((vehicleused.postalcode)::text) =
lower((postalcoderegioncountycity.postalcode)::text))"
" -> Nested Loop (cost=101.81..37776.78
rows=11887 width=10) (actual time=1.172..9876.586 rows=382528 loops=1)"
" -> Bitmap Heap Scan on
datasetgroupyearmakemodel (cost=101.81..948.81 rows=5360 width=6)
(actual time=0.988..17.800 rows=5377 loops=1)"
" Recheck Cond: (datasetgroupid =
3)"
" -> Bitmap Index Scan on
datasetgroupyearmakemodel_i04 (cost=0.00..100.47 rows=5360 width=0)
(actual time=0.830..0.830 rows=5377 loops=1)"
" Index Cond: (datasetgroupid
= 3)"
" -> Index Scan using vehicleused_i10 on
vehicleused (cost=0.00..6.85 rows=1 width=12) (actual time=0.049..1.775
rows=71 loops=5377)"
" Index Cond:
((vehicleused.vehiclemodelid = datasetgroupyearmakemodel.vehiclemodelid)
AND (vehicleused.vehiclemakeid =
datasetgroupyearmakemodel.vehiclemakeid) AND (vehicleused.vehicleyear =
datasetgroupyearmakemodel.vehicleyear))"
" Filter: ((vehicleused.datasetid
<> 113) AND (vehicleused.productgroupid <> 13))"
" -> Hash (cost=308.93..308.93 rows=1416
width=6) (actual time=2.738..2.738 rows=1435 loops=1)"
" -> Bitmap Heap Scan on
postalcoderegioncountycity (cost=27.23..308.93 rows=1416 width=6)
(actual time=0.222..0.955 rows=1435 loops=1)"
" Recheck Cond: (regionid = 36)"
" -> Bitmap Index Scan on
postalcoderegioncountycity_i05 (cost=0.00..26.87 rows=1416 width=0)
(actual time=0.202..0.202 rows=1435 loops=1)"
" Index Cond: (regionid =
36)"
" -> Hash (cost=4.61..4.61 rows=261 width=10)
(actual time=0.307..0.307 rows=261 loops=1)"
" -> Seq Scan on vehiclemake (cost=0.00..4.61
rows=261 width=10) (actual time=0.033..0.154 rows=261 loops=1)"
"Total runtime: 10676.058 ms"
Pam Ozer
Data Architect
pozer@automotive.com <mailto:pozer@automotive.com>
tel. 949.705.3468
Source Interlink Media
1733 Alton Pkwy Suite 100, Irvine, CA 92606
www.simautomotive.com <http://www.simautomotive.com;
Confidentiality Notice- This electronic communication, and all
information herein, including files attached hereto, is private, and is
the property of the sender. This communication is intended only for the
use of the individual or entity named above. If you are not the intended
recipient, you are hereby notified that any disclosure of; dissemination
of; distribution of; copying of; or, taking any action in reliance upon
this communication, is strictly prohibited. If you have received this
communication in error, please immediately notify us by telephone,
(949)-705-3000, and destroy all copies of this communication. Thank you.
Attachments:
[image/jpeg] image001.jpg (3.7K, ../../C3CA20D47A2DF94FBF9CF0AE8D964CC412438A@mail-001.corp.automotive.com/3-image001.jpg)
download | view image
[image/jpeg] image002.jpg (346B, ../../C3CA20D47A2DF94FBF9CF0AE8D964CC412438A@mail-001.corp.automotive.com/4-image002.jpg)
download | view image
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Slow Query
@ 2010-08-27 00:18 Scott Marlowe <scott.marlowe@gmail.com>
parent: Ozer, Pam <pozer@automotive.com>
3 siblings, 0 replies; 112+ messages in thread
From: Scott Marlowe @ 2010-08-27 00:18 UTC (permalink / raw)
To: Ozer, Pam <pozer@automotive.com>; +Cc: pgsql-performance
On Thu, Aug 26, 2010 at 6:03 PM, Ozer, Pam <pozer@automotive.com> wrote:
>
> I am new to Postgres and I am trying to understand the Explain Analyze so I can tune the following query. I run the same query using mysql and it takes less than 50ms. I run it on postgres and it takes 10 seconds. I feel like I am missing something very obvious. (VehicleUsed is a big table over 750,000records) and datasetgroupyearmakemodel has 150000 records.
>
> It looks like the cost is highest in the Hash Join on Postalcode. Am I reading this correctly.? I do have indexes on the lower(postalcode) in both tables. Why wouldn’t be using the index?
No, it's spending most of its time here:
> " -> Nested Loop (cost=101.81..37776.78 rows=11887 width=10) (actual time=1.172..9876.586 rows=382528 loops=1)"
Note that it expects 11,887 rows but gets 382k rows.
Try turning up default stats target and running analyze again and see
how it runs.
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Slow Query
@ 2010-08-27 00:19 Bob Lunney <bob_lunney@yahoo.com>
parent: Ozer, Pam <pozer@automotive.com>
3 siblings, 0 replies; 112+ messages in thread
From: Bob Lunney @ 2010-08-27 00:19 UTC (permalink / raw)
To: pgsql-performance@postgresql.org, PamOzer <pozer@automotive.com>
We need more information than that, like:
What version of PostgreSQL?
What does the hardware look like?
What does the disk and tablespace layout look like?
How are your configuration variables set?
Other than that, are the statistics up to date on the VehicleMake table?
Bob Lunney
--- On Thu, 8/26/10, Ozer, Pam <pozer@automotive.com> wrote:
From: Ozer, Pam <pozer@automotive.com>
Subject: [PERFORM] Slow Query
To: pgsql-performance@postgresql.org
Date: Thursday, August 26, 2010, 8:03 PM
I am new to Postgres and I am trying to understand the
Explain Analyze so I can tune the following query. I run the same query using
mysql and it takes less than 50ms. I run it on postgres and it takes 10
seconds. I feel like I am missing something very obvious. (VehicleUsed is a big
table over 750,000records) and datasetgroupyearmakemodel has 150000 records.
It looks like the cost is highest in the Hash Join on
Postalcode. Am I reading this correctly.? I do have indexes on the
lower(postalcode) in both tables. Why wouldn’t be using the index? Thanks
in advance for any help.
Here is my query:
select distinct VehicleMake.VehicleMake
from VehicleUsed
inner join PostalCodeRegionCountyCity on ( lower (
VehicleUsed.PostalCode ) = lower ( PostalCodeRegionCountyCity.PostalCode ) )
INNER JOIN DATASETGROUPYEARMAKEMODEL ON (
VEHICLEUSED.VEHICLEYEAR = DATASETGROUPYEARMAKEMODEL.VEHICLEYEAR )
AND ( VEHICLEUSED.VEHICLEMAKEID =
DATASETGROUPYEARMAKEMODEL.VEHICLEMAKEID )
AND ( VEHICLEUSED.VEHICLEMODELID =
DATASETGROUPYEARMAKEMODEL.VEHICLEMODELID )
inner join VehicleMake on ( VehicleUsed.VehicleMakeId =
VehicleMake.VehicleMakeId )
where
( DatasetGroupYearMakeModel.DatasetGroupId = 3 ) and
( VehicleUsed.DatasetId <> 113 )
and ( VehicleUsed.ProductGroupId <> 13 )
and ( PostalCodeRegionCountyCity.RegionId = 36 )
order by VehicleMake.VehicleMake
limit 500000
Here is the explain analyze
"Limit (cost=38292.53..38293.19 rows=261 width=8)
(actual time=10675.857..10675.892 rows=42 loops=1)"
" -> Sort (cost=38292.53..38293.19 rows=261
width=8) (actual time=10675.855..10675.868 rows=42 loops=1)"
" Sort Key: vehiclemake.vehiclemake"
" Sort Method: quicksort Memory: 18kB"
" -> HashAggregate (cost=38279.45..38282.06
rows=261 width=8) (actual time=10675.710..10675.728 rows=42 loops=1)"
" -> Hash Join (cost=436.31..38270.51
rows=3576 width=8) (actual time=4.471..10658.291 rows=10425 loops=1)"
" Hash Cond:
(vehicleused.vehiclemakeid = vehiclemake.vehiclemakeid)"
" -> Hash Join
(cost=428.43..38213.47 rows=3576 width=4) (actual time=4.152..10639.742
rows=10425 loops=1)"
" Hash Cond:
(lower((vehicleused.postalcode)::text) =
lower((postalcoderegioncountycity.postalcode)::text))"
" -> Nested Loop
(cost=101.81..37776.78 rows=11887 width=10) (actual time=1.172..9876.586
rows=382528 loops=1)"
" -> Bitmap Heap
Scan on datasetgroupyearmakemodel (cost=101.81..948.81 rows=5360 width=6)
(actual time=0.988..17.800 rows=5377 loops=1)"
" Recheck Cond:
(datasetgroupid = 3)"
" -> Bitmap
Index Scan on datasetgroupyearmakemodel_i04 (cost=0.00..100.47 rows=5360
width=0) (actual time=0.830..0.830 rows=5377 loops=1)"
" Index
Cond: (datasetgroupid = 3)"
" -> Index Scan
using vehicleused_i10 on vehicleused (cost=0.00..6.85 rows=1 width=12) (actual
time=0.049..1.775 rows=71 loops=5377)"
" Index Cond:
((vehicleused.vehiclemodelid = datasetgroupyearmakemodel.vehiclemodelid) AND
(vehicleused.vehiclemakeid = datasetgroupyearmakemodel.vehiclemakeid) AND
(vehicleused.vehicleyear = datasetgroupyearmakemodel.vehicleyear))"
" Filter:
((vehicleused.datasetid <> 113) AND (vehicleused.productgroupid <>
13))"
" -> Hash
(cost=308.93..308.93 rows=1416 width=6) (actual time=2.738..2.738 rows=1435
loops=1)"
" -> Bitmap Heap
Scan on postalcoderegioncountycity (cost=27.23..308.93 rows=1416 width=6)
(actual time=0.222..0.955 rows=1435 loops=1)"
" Recheck Cond:
(regionid = 36)"
" -> Bitmap
Index Scan on postalcoderegioncountycity_i05 (cost=0.00..26.87 rows=1416
width=0) (actual time=0.202..0.202 rows=1435 loops=1)"
" Index
Cond: (regionid = 36)"
" -> Hash (cost=4.61..4.61
rows=261 width=10) (actual time=0.307..0.307 rows=261 loops=1)"
" -> Seq Scan on
vehiclemake (cost=0.00..4.61 rows=261 width=10) (actual time=0.033..0.154
rows=261 loops=1)"
"Total runtime: 10676.058 ms"
Pam Ozer
Data Architect
pozer@automotive.com
tel. 949.705.3468
Source Interlink Media
1733 Alton Pkwy Suite 100, Irvine, CA 92606
www.simautomotive.com
Confidentiality Notice- This electronic communication, and all
information herein, including files attached hereto, is private, and is the
property of the sender. This communication is intended only for the use of the
individual or entity named above. If you are not the intended recipient, you
are hereby notified that any disclosure of; dissemination of; distribution of;
copying of; or, taking any action in reliance upon this communication, is
strictly prohibited. If you have received this communication in error, please
immediately notify us by telephone, (949)-705-3000, and destroy all copies of
this communication. Thank you.
Attachments:
[image/jpeg] image001.jpg (3.7K, ../../599616.8843.qm@web39705.mail.mud.yahoo.com/3-image001.jpg)
download | view image
[image/jpeg] image002.jpg (346B, ../../599616.8843.qm@web39705.mail.mud.yahoo.com/4-image002.jpg)
download | view image
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Slow Query
@ 2010-08-27 14:17 Kevin Grittner <Kevin.Grittner@wicourts.gov>
parent: Ozer, Pam <pozer@automotive.com>
3 siblings, 0 replies; 112+ messages in thread
From: Kevin Grittner @ 2010-08-27 14:17 UTC (permalink / raw)
To: Pam Ozer <pozer@automotive.com>; pgsql-performance
"Ozer, Pam" <pozer@automotive.com> wrote:
> I am new to Postgres and I am trying to understand the Explain
> Analyze so I can tune the following query.
Fair enough. If you wanted advice from others on how to tune the
query, you would need to provide additional information, as
described here:
http://wiki.postgresql.org/wiki/SlowQueryQuestions
> It looks like the cost is highest in the Hash Join on Postalcode.
> Am I reading this correctly.?
No.
Here's how I read it:
It's scanning the datasetgroupyearmakemodel_i04 index for
datasetgroupid = 3, and finding 5377 index entries in 1.775 ms.
Then it read the related tuples (in the order they exist in the
table's heap space). Added to the index scan, that brings us to
17.8 ms.
For each of the preceding rows, it uses the vehicleused_i10 index to
find matching vehicleused rows. On average it finds 71 rows in
1.775 ms, but it does this 5377 times, which brings us to 382528
rows in 9876.586 ms.
Independently of the above, it uses the
postalcoderegioncountycity_i05 index to find
postalcoderegioncountycity rows where regionid = 36, and puts them
into a hash table. It takes 2.738 ms to hash the 1435 matching
rows. Probing this hash for each of the 382528 rows leaves us with
10425 rows, and brings the run time to 10639.742 ms.
Independently of the above, the vehiclemake table is sequentially
scanned to create a hash table with 261 rows. That takes 0.307 ms.
Probing that for each of the 10425 rows doesn't eliminate anything
and bring the run time to 10658.291 ms.
Since DISTINCT was specified, the results of the preceding are fed
into a hash table, leaving 42 distinct values. Run time has now
reached 10675.728 ms.
These 42 rows are sorted, based on the ORDER BY clause, bringing us
to 10675.868 ms.
The output of the sort goes through a filter which will cut off
output after 500000 rows, per the LIMIT clause, bringing run time to
10675.892 ms.
> I do have indexes on the lower(postalcode)
> in both tables. Why wouldn't be using the index?
Either it doesn't think it can use that index for some reason (e.g.,
one column is char(n) and the other is varchar(n)), or it thinks
that using that index will be slower. The latter could be due to
bad statistics or improper configuration. Without more information,
it's hard to guess why it thinks that or whether it's right.
I suggest you read the SlowQueryQuestions link and post with more
information.
I will offer the observation that the numbers above suggest a fair
amount of the data for this query came from cache, and unless you
have tuned your database's costing factors to allow it to expect
that, it's not surprising that it makes bad choices. Just as an
experiment, try running these statements in a connection before your
query on that same connection -- if they work out well, it might
make sense to make similar changes to the postgresql.conf file:
set effective_cache_size = '3GB'; -- or 1GB less than total RAM
set seq_page_cost = '0.1';
set random_page_cost = '0.1';
-Kevin
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Slow Query
@ 2010-08-27 17:30 Greg Smith <greg@2ndquadrant.com>
parent: Ozer, Pam <pozer@automotive.com>
3 siblings, 1 reply; 112+ messages in thread
From: Greg Smith @ 2010-08-27 17:30 UTC (permalink / raw)
To: Ozer, Pam <pozer@automotive.com>; +Cc: pgsql-performance
Ozer, Pam wrote:
>
> I am new to Postgres and I am trying to understand the Explain Analyze
> so I can tune the following query.
>
You should try http://explain.depesz.com/ , where you can post query
plans like this and see where the time is going in a very easy to work
with form. It will highlight row estimation mistakes for you too.
--
Greg Smith 2ndQuadrant US Baltimore, MD
PostgreSQL Training, Services and Support
greg@2ndQuadrant.com www.2ndQuadrant.us
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Slow Query
@ 2010-08-27 17:44 Ozer, Pam <pozer@automotive.com>
parent: Greg Smith <greg@2ndquadrant.com>
0 siblings, 0 replies; 112+ messages in thread
From: Ozer, Pam @ 2010-08-27 17:44 UTC (permalink / raw)
To: Greg Smith <greg@2ndquadrant.com>; +Cc: pgsql-performance
Thank you all for your help. I will also get the rest of the
information so it will be more clear.
From: Greg Smith [mailto:greg@2ndquadrant.com]
Sent: Friday, August 27, 2010 10:31 AM
To: Ozer, Pam
Cc: pgsql-performance@postgresql.org
Subject: Re: [PERFORM] Slow Query
Ozer, Pam wrote:
I am new to Postgres and I am trying to understand the Explain Analyze
so I can tune the following query.
You should try http://explain.depesz.com/ , where you can post query
plans like this and see where the time is going in a very easy to work
with form. It will highlight row estimation mistakes for you too.
--
Greg Smith 2ndQuadrant US Baltimore, MD
PostgreSQL Training, Services and Support
greg@2ndQuadrant.com www.2ndQuadrant.us
^ permalink raw reply [nested|flat] 112+ messages in thread
* Slow query
@ 2014-03-07 06:05 Bikram Kesari Naik <bikram.naik@onmobile.com>
0 siblings, 1 reply; 112+ messages in thread
From: Bikram Kesari Naik @ 2014-03-07 06:05 UTC (permalink / raw)
To: pgsql-performance; +Cc: Sukruth S <sukruth@onmobile.com>
Hi,
I have a view which joins multiple tables to give me a result. It takes more than a minute to give me the result on psql prompt when I select all the data from that view.
The single CPU which is used to run this query is utilized 100%.Even if I fire a count(*) it takes 10 Sec. I wanted to know if there is anything we can do to speedup this query below 1 sec.
CREATE OR REPLACE VIEW wh_rbtmapdetails_test AS
SELECT map_tab.binarymapid,
map_tab.whsongid,
map_tab.binaryid,
map_tab.previewbinaryid,
map_tab.created,
map_tab.createdby,
map_tab.lastmodified,
map_tab.lastmodifiedby,
map_tab.stateon,
map_tab.stateby,
map_tab.statename,
map_tab.statereason,
bin_tab.filepath,
bin_tab.filesizeinbytes,
bin_tab.contenttypename,
bin_tab.fileextension,
bin_tab.filepath AS previewfilepath,
bin_tab.contenttypename AS previewcontenttypename,
bin_tab.fileextension AS previewfileextension,
bin_tab.lastmodified AS binlastmodified,
bin_tab.created AS bincreated,
md_tab.whsongname,
md_tab.whsongnamerx,
md_tab.whmoviename,
md_tab.whmovienamerx,
md_tab.languagename,
md_tab.contentproviderid,
md_tab.rightsbodyid,
md_tab.labelid,
md_tab.isrc,
md_tab.keywords,
md_tab.cpcontentid,
md_tab.songreleasedate,
md_tab.moviereleasedate,
md_tab.actor,
md_tab.singer,
md_tab.musicdirector,
md_tab.moviedirector,
md_tab.movieproducer,
md_tab.rightsbodyname,
md_tab.labelname,
md_tab.contentprovidername,
md_tab.categoryid,
md_tab.categoryname,
md_tab.subcategoryname,
md_tab.genrename,
md_tab.promocode,
md_tab.cptransferdate,
md_tab.lastmodified AS mdlastmodified,
md_tab.created AS mdcreated,
map_tab.holdercontentsubtypeid,
NULL::unknown AS holdercontentsubtypename,
md_tab.statename AS metadatastatename,
md_tab.statereason AS metadatastatereason,
bin_tab.statename AS binarystatename,
bin_tab.statereason AS binarystatereason,
md_tab.isbranded,
md_tab.brandname,
md_tab.aliaspromocode,
md_tab.songreleaseyear,
md_tab.moviereleaseyear,
md_tab.songid,
map_tab.holderstartdate,
map_tab.holderenddate,
md_tab.comments,
md_tab.iprs,
md_tab.lyricist,
md_tab.workssociety,
md_tab.publisher,
md_tab.iswc,
songartwork.stateon as artworkstateon,
'' AS airtelvcode,
'' AS airtelccode
FROM songbinarymap map_tab
inner join wh_songmetadatadetails_sukruth md_tab on map_tab.whsongid = md_tab.whsongid
inner join songbinarywarehouse_rbt bin_tab on map_tab.binaryid = bin_tab.binaryid
inner join contentprovider cp_tab on md_tab.contentproviderid = cp_tab.contentproviderid
left join songartwork on songartwork.whsongid = map_tab.whsongid
WHERE cp_tab.hide <> 1;
--##############################################################################################################################
atlantisindia=# Select count(*) from wh_songmetadatadetails_sukruth;
count
---------
2756891
atlantisindia=# Select count(*) from songbinarywarehouse_rbt;
count
---------
3507188
atlantisindia=# Select count(*) from contentprovider;
count
-------
446
atlantisindia=# Select count(*) from songartwork;
count
--------
292457
atlantisindia=# Select count(*) from songbinarymap;
count
---------
3460677
Objects used in the query:
wh_songmetadatadetails_sukruth -- view
songbinarywarehouse_rbt -- Table
songartwork -- Table
songbinarymap -- Table
/*
CREATE OR REPLACE VIEW wh_songmetadatadetails_sukruth AS
SELECT md_tab.whsongid,
md_tab.whsongname,
md_tab.whsongnamerx,
md_tab.whmoviename,
md_tab.whmovienamerx,
md_tab.languagename,
md_tab.contentproviderid,
md_tab.rightsbodyid,
md_tab.labelid,
md_tab.isrc,
md_tab.songreleasedate,
md_tab.moviereleasedate,
md_tab.actor,
md_tab.singer,
md_tab.musicdirector,
md_tab.moviedirector,
md_tab.movieproducer,
md_tab.keywords,
md_tab.cpcontentid,
md_tab.created,
md_tab.createdby,
md_tab.lastmodified,
md_tab.lastmodifiedby,
md_tab.stateon,
md_tab.stateby,
md_tab.statename,
md_tab.statereason,
md_tab.comments,
md_tab.oldcategoryname,
md_tab.oldsubcategoryname,
md_tab.oldgenrename,
rightsbody.rightsbodyname,
label.labelname,
cp_tab.contentprovidername,
cp_tab.alias_contentproviderid,
md_tab.promocode,
md_tab.isbranded,
md_tab.brandname,
'now'::text::timestamp without time zone - md_tab.stateon AS sysdatestateondiff,
md_tab.aliaspromocode,
md_tab.songreleaseyear,
md_tab.moviereleaseyear,
md_tab.costrc,
md_tab.categoryid,
cpcategoryforselect.categoryname,
md_tab.subcategoryname,
md_tab.genrename,
md_tab.metadatacorrection,
md_tab.songid,
songartwork.artworkbinaryid_1,
md_tab.iprs,
md_tab.lyricist,
md_tab.workssociety,
md_tab.publisher,
md_tab.iswc,
md_tab.umdbid,
md_tab.umdbmodified,
md_tab.cptransferdate,
CAST('' as varchar(1)) airtelvcode,
cast('' as varchar(1)) airtelccode
FROM songmetadatawarehouse md_tab
inner join contentprovider cp_tab on md_tab.contentproviderid::text = cp_tab.contentproviderid::text
left join songartwork on songartwork.whsongid::text = md_tab.whsongid::text
left join cpcategoryforselect on cpcategoryforselect.categoryid::text = md_tab.categoryid::text
left join label on label.labelid::text = md_tab.labelid::text
left join rightsbody on rightsbody.rightsbodyid::text = md_tab.rightsbodyid::text
WHERE cp_tab.hide <> 1;
*/
--########################################### Exolain Plan ######################################################
explain analyze select * from wh_rbtmapdetails_test;
QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------
Hash Left Join (cost=597862.86..1042112.62 rows=3322444 width=1714) (actual time=11015.390..29802.769 rows=2615033 loops=1)
Hash Cond: ((map_tab.whsongid)::text = (songartwork.whsongid)::text)
-> Hash Join (cost=581788.58..976448.68 rows=3322444 width=1706) (actual time=10828.076..25597.518 rows=2615033 loops=1)
Hash Cond: ((md_tab.contentproviderid)::text = (cp_tab.contentproviderid)::text)
-> Hash Left Join (cost=581761.54..930481.44 rows=3390870 width=1734) (actual time=10827.746..24577.167 rows=2615033 loops=1)
Hash Cond: ((md_tab.categoryid)::text = (cpcategoryforselect.categoryid)::text)
-> Hash Join (cost=581759.59..883858.42 rows=3390870 width=1702) (actual time=10827.690..23549.201 rows=2615033 loops=1)
Hash Cond: ((map_tab.whsongid)::text = (md_tab.whsongid)::text)
-> Hash Join (cost=192309.86..425892.93 rows=3460705 width=538) (actual time=2883.529..10130.740 rows=2802042 loops=1)
Hash Cond: ((bin_tab.binaryid)::text = (map_tab.binaryid)::text)
-> Seq Scan on songbinarywarehouse_rbt bin_tab (cost=0.00..159538.54 rows=3505554 width=135) (actual time=0.002..362.037 rows=3507188 loops
=1)
-> Hash (cost=149051.05..149051.05 rows=3460705 width=436) (actual time=2881.792..2881.792 rows=3460677 loops=1)
Buckets: 524288 Batches: 1 Memory Usage: 888911kB
-> Seq Scan on songbinarymap map_tab (cost=0.00..149051.05 rows=3460705 width=436) (actual time=0.004..1129.036 rows=3460677 loops=1)
-> Hash (cost=353389.46..353389.46 rows=2884822 width=1197) (actual time=7941.814..7941.814 rows=2756891 loops=1)
Buckets: 524288 Batches: 1 Memory Usage: 1339511kB
-> Hash Left Join (cost=307.32..353389.46 rows=2884822 width=1197) (actual time=2.585..4859.733 rows=2756891 loops=1)
Hash Cond: ((md_tab.rightsbodyid)::text = (rightsbody.rightsbodyid)::text)
-> Hash Left Join (cost=234.82..313650.65 rows=2884822 width=1183) (actual time=1.978..3756.064 rows=2756891 loops=1)
Hash Cond: ((md_tab.labelid)::text = (label.labelid)::text)
-> Hash Join (cost=27.04..262958.49 rows=2884822 width=1167) (actual time=0.257..2703.354 rows=2756891 loops=1)
Hash Cond: ((md_tab.contentproviderid)::text = (cp_tab_1.contentproviderid)::text)
-> Seq Scan on songmetadatawarehouse md_tab (cost=0.00..223042.35 rows=2944235 width=1125) (actual time=0.002..363.384 ro
ws=2944240 loops=1)
-> Hash (cost=21.57..21.57 rows=437 width=42) (actual time=0.239..0.239 rows=431 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 32kB
-> Seq Scan on contentprovider cp_tab_1 (cost=0.00..21.57 rows=437 width=42) (actual time=0.003..0.141 rows=431 loo
ps=1)
Filter: (hide <> 1)
Rows Removed by Filter: 15
-> Hash (cost=160.68..160.68 rows=3768 width=49) (actual time=1.708..1.708 rows=3768 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 301kB
-> Seq Scan on label (cost=0.00..160.68 rows=3768 width=49) (actual time=0.003..0.797 rows=3768 loops=1)
-> Hash (cost=55.00..55.00 rows=1400 width=47) (actual time=0.596..0.596 rows=1400 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 109kB
-> Seq Scan on rightsbody (cost=0.00..55.00 rows=1400 width=47) (actual time=0.002..0.258 rows=1400 loops=1)
-> Hash (cost=1.70..1.70 rows=20 width=65) (actual time=0.033..0.033 rows=20 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 2kB
-> Subquery Scan on cpcategoryforselect (cost=0.00..1.70 rows=20 width=65) (actual time=0.013..0.025 rows=20 loops=1)
-> Seq Scan on cpcategory (cost=0.00..1.50 rows=20 width=53) (actual time=0.012..0.022 rows=20 loops=1)
-> Hash (cost=21.57..21.57 rows=437 width=28) (actual time=0.306..0.306 rows=431 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 26kB
-> Seq Scan on contentprovider cp_tab (cost=0.00..21.57 rows=437 width=28) (actual time=0.027..0.212 rows=431 loops=1)
Filter: (hide <> 1)
Rows Removed by Filter: 15
-> Hash (cost=12418.57..12418.57 rows=292457 width=41) (actual time=187.181..187.181 rows=292457 loops=1)
Buckets: 32768 Batches: 1 Memory Usage: 20849kB
-> Seq Scan on songartwork (cost=0.00..12418.57 rows=292457 width=41) (actual time=0.024..102.322 rows=292457 loops=1)
Total runtime: 29966.975 ms
(47 rows)
--##############################################################################################################################
System Info:
# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Thread(s) per core: 2
Core(s) per socket: 4
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 45
Stepping: 7
CPU MHz: 1200.000
BogoMIPS: 6599.09
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 10240K
NUMA node0 CPU(s): 0-3,8-11
NUMA node1 CPU(s): 4-7,12-15
--##############################################################################################################################
#free
total used free shared buffers cached
Mem: 65932076 34444056 31488020 0 307152 31947732
-/+ buffers/cache: 2189172 63742904
Swap: 33038328 352928 32685400
--##############################################################################################################################
[root@localhost ~]# top
top - 00:42:59 up 113 days, 8:58, 3 users, load average: 0.08, 0.02, 0.01
Tasks: 568 total, 2 running, 565 sleeping, 0 stopped, 1 zombie
Cpu(s): 5.9%us, 0.5%sy, 0.0%ni, 93.6%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 65932076k total, 40638080k used, 25293996k free, 311728k buffers
Swap: 33038328k total, 352844k used, 32685484k free, 35244776k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
36725 postgres 20 0 17.0g 5.9g 3.5g R 99.8 9.4 0:35.91 postmaster
30038 oracle -2 0 19.1g 16m 14m S 1.3 0.0 2:29.59 oracle
41270 root 20 0 15436 1640 956 R 0.7 0.0 0:00.12 top
30141 oracle 20 0 19.1g 16m 14m S 0.3 0.0 0:14.72 oracle
1 root 20 0 19356 796 584 S 0.0 0.0 3:01.96 init
2 root 20 0 0 0 0 S 0.0 0.0 0:04.82 kthreadd
3 root RT 0 0 0 0 S 0.0 0.0 0:15.46 migration/0
4 root 20 0 0 0 0 S 0.0 0.0 0:16.53 ksoftirqd/0
5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
6 root RT 0 0 0 0 S 0.0 0.0 0:09.89 watchdog/0
7 root RT 0 0 0 0 S 0.0 0.0 0:19.86 migration/1
8 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/1
9 root 20 0 0 0 0 S 0.0 0.0 0:21.03 ksoftirqd/1
10 root RT 0 0 0 0 S 0.0 0.0 0:08.66 watchdog/1
11 root RT 0 0 0 0 S 0.0 0.0 0:05.25 migration/2
12 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/2
13 root 20 0 0 0 0 S 0.0 0.0 0:04.83 ksoftirqd/2
14 root RT 0 0 0 0 S 0.0 0.0 0:08.87 watchdog/2
15 root RT 0 0 0 0 S 0.0 0.0 0:04.10 migration/3
16 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/3
17 root 20 0 0 0 0 S 0.0 0.0 0:03.56 ksoftirqd/3
18 root RT 0 0 0 0 S 0.0 0.0 0:08.68 watchdog/3
19 root RT 0 0 0 0 S 0.0 0.0 0:17.71 migration/4
20 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/4
21 root 20 0 0 0 0 S 0.0 0.0 0:30.47 ksoftirqd/4
22 root RT 0 0 0 0 S 0.0 0.0 0:11.47 watchdog/4
23 root RT 0 0 0 0 S 0.0 0.0 0:25.19 migration/5
24 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/5
25 root 20 0 0 0 0 S 0.0 0.0 0:35.14 ksoftirqd/5
26 root RT 0 0 0 0 S 0.0 0.0 0:10.34 watchdog/5
27 root RT 0 0 0 0 S 0.0 0.0 0:06.08 migration/6
28 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/6
29 root 20 0 0 0 0 S 0.0 0.0 0:14.71 ksoftirqd/6
30 root RT 0 0 0 0 S 0.0 0.0 0:09.54 watchdog/6
31 root RT 0 0 0 0 S 0.0 0.0 0:02.36 migration/7
32 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/7
33 root 20 0 0 0 0 S 0.0 0.0 0:06.70 ksoftirqd/7
--##############################################################################################################################
[root@localhost ~]# vmstat -a 1
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free inact active si so bi bo in cs us sy id wa st
1 0 352844 19137128 16133388 29410720 0 0 2 8 0 0 0 0 100 0 0
1 0 352844 20462344 16133388 28085988 0 0 0 16 2524 19813 6 1 92 0 0
1 0 352844 20984756 16133412 27564440 0 0 0 32 2276 2711 4 3 94 0 0
1 0 352844 20984524 16133412 27564760 0 0 0 0 2197 2572 6 0 94 0 0
2 0 352844 20984544 16133412 27564760 0 0 0 20 2231 2634 6 0 94 0 0
3 0 352844 20983176 16133416 27564752 0 0 0 216 2342 2690 6 0 93 0 0
2 0 352844 20983308 16133416 27564808 0 0 0 0 2227 2634 6 0 94 0 0
1 0 352844 20983556 16133416 27564808 0 0 0 0 2227 2597 6 0 94 0 0
1 0 352844 20983664 16133416 27564808 0 0 0 32 2263 2643 6 0 93 0 0
1 0 352844 20983400 16133416 27564920 0 0 0 4 2217 2616 6 0 94 0 0
2 0 352844 20987048 16133428 27560248 0 0 0 108 3092 3163 7 1 92 0 0
3 0 352844 20987172 16133416 27560248 0 0 0 32 2316 2716 6 0 94 0 0
2 0 352844 20987188 16133412 27559624 0 0 0 0 2226 2651 6 0 94 0 0
1 0 352844 20987304 16133412 27559624 0 0 0 0 2198 2614 6 0 94 0 0
1 0 352844 20983808 16133416 27563632 0 0 0 32 2422 2839 7 0 93 0 0
1 0 352844 20987420 16133416 27559956 0 0 0 0 2307 2638 6 0 94 0 0
3 0 352844 20987712 16133416 27559700 0 0 0 140 2278 2704 6 0 94 0 0
2 0 352844 20987712 16133412 27559752 0 0 0 88 2306 2706 6 0 94 0 0
2 0 352844 20987844 16133416 27559660 0 0 0 0 2233 2600 6 0 94 0 0
1 0 352844 20988000 16133416 27559660 0 0 0 0 2202 2582 6 0 94 0 0
1 0 352844 20988124 16133416 27559664 0 0 0 44 2278 2654 6 0 94 0 0
1 0 352844 20988240 16133416 27559664 0 0 0 0 2193 2571 6 0 94 0 0
2 0 352844 20988248 16133416 27559664 0 0 0 64 2230 2656 6 0 94 0 0
3 0 352844 20988356 16133416 27559668 0 0 0 32 2275 2637 6 0 94 0 0
2 0 352844 20988668 16133416 27559668 0 0 0 12 2233 2664 6 0 94 0 0
1 0 352844 20988660 16133420 27559672 0 0 0 0 2238 2614 6 0 94 0 0
1 0 352844 20988784 16133420 27559672 0 0 0 44 2243 2658 6 0 94 0 0
2 0 352844 20988040 16133420 27560036 0 0 0 12 2256 2664 6 0 93 0 0
1 0 352844 20987312 16133416 27560752 0 0 0 88 2314 2706 6 0 93 0 0
1 0 352844 20986600 16133468 27561040 0 0 0 64 2343 2688 6 0 94 0 0
2 0 352844 20987212 16133460 27560668 0 0 0 12 2255 2658 6 0 94 0 0
2 0 352844 20987336 16133460 27560724 0 0 0 12 2211 2593 6 0 94 0 0
2 0 352844 20987352 16133464 27560604 0 0 0 32 2295 2732 6 0 94 0 0
1 0 352844 20987148 16133464 27560832 0 0 0 0 3041 3197 7 1 92 0 0
1 0 352844 20986848 16133464 27560728 0 0 0 108 2267 2670 6 0 94 0 0
1 0 352844 20986948 16133468 27560608 0 0 0 32 2261 2652 6 0 94 0 0
2 0 352844 20987064 16133468 27560608 0 0 0 0 2208 2610 6 0 94 0 0
2 0 352844 20987064 16133468 27560608 0 0 0 0 2192 2559 6 0 94 0 0
1 0 352844 20987236 16133468 27560608 0 0 0 32 2256 2630 6 0 94 0 0
1 0 352844 20987368 16133468 27560608 0 0 0 0 2223 2624 6 0 94 0 0
1 0 352844 20980036 16133552 27565904 0 0 0 208 2457 2855 7 0 93 0 0
--##############################################################################################################################
[root@localhost ~]# iostat -xn 1
Linux 2.6.32-358.el6.x86_64 (localhost.localdomain) 03/07/2014 _x86_64_ (16 CPU)
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 0.05 28.93 0.44 4.63 65.60 268.49 65.88 0.04 8.28 0.32 0.16
dm-0 0.00 0.00 0.05 0.09 0.37 0.74 8.00 0.00 6.07 0.16 0.00
dm-1 0.00 0.00 0.08 18.97 4.65 151.78 8.21 0.24 12.71 0.06 0.11
dm-2 0.00 0.00 0.09 5.97 11.55 47.77 9.78 0.43 71.51 0.05 0.03
dm-3 0.00 0.00 0.27 8.52 49.04 68.19 13.33 0.03 3.81 0.05 0.04
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 12.00 0.00 14.00 0.00 208.00 14.86 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 0.00 26.00 0.00 208.00 8.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 0.00 0.00 1.00 0.00 8.00 8.00 0.00 3.00 3.00 0.30
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 0.00 1.00 0.00 8.00 8.00 0.00 3.00 3.00 0.30
dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 24.00 0.00 11.00 0.00 280.00 25.45 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 0.00 35.00 0.00 280.00 8.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 0.00 0.00 1.00 0.00 8.00 8.00 0.00 3.00 3.00 0.30
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 0.00 1.00 0.00 8.00 8.00 0.00 3.00 3.00 0.30
dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 6.00 0.00 5.00 0.00 88.00 17.60 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 0.00 8.00 0.00 64.00 8.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 0.00 3.00 0.00 24.00 8.00 0.00 0.00 0.00 0.00
dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 1.00 0.00 1.00 0.00 16.00 16.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 0.00 2.00 0.00 16.00 8.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Thanks,
Bikram
________________________________
DISCLAIMER: The information in this message is confidential and may be legally privileged. It is intended solely for the addressee. Access to this message by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, or distribution of the message, or any action or omission taken by you in reliance on it, is prohibited and may be unlawful. Please immediately contact the sender if you have received this message in error. Further, this e-mail may contain viruses and all reasonable precaution to minimize the risk arising there from is taken by OnMobile. OnMobile is not liable for any damage sustained by you as a result of any virus in this e-mail. All applicable virus checks should be carried out by you before opening this e-mail or any attachment thereto.
Thank you - OnMobile Global Limited.
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Slow query
@ 2014-03-07 06:23 David Johnston <polobo@yahoo.com>
parent: Bikram Kesari Naik <bikram.naik@onmobile.com>
0 siblings, 1 reply; 112+ messages in thread
From: David Johnston @ 2014-03-07 06:23 UTC (permalink / raw)
To: pgsql-performance
Bikram Kesari Naik wrote
> Hi,
>
> I have a view which joins multiple tables to give me a result. It takes
> more than a minute to give me the result on psql prompt when I select all
> the data from that view.
> The single CPU which is used to run this query is utilized 100%.Even if I
> fire a count(*) it takes 10 Sec. I wanted to know if there is anything we
> can do to speedup this query below 1 sec.
In all likelihood you need to index your foreign keys, and possibly other
fields, but as you haven't provided table and index definitions it is hard
to say for sure.
Idepeing on how many rows are hidden I'm not sure an unqualified query on
this view can run in 1/60th the time even with indexes present - the
sequential scans are efficient if the proportion of the tables being
returned is high.
David J.
--
View this message in context: http://postgresql.1045698.n5.nabble.com/Slow-query-tp5795077p5795079.html
Sent from the PostgreSQL - performance mailing list archive at Nabble.com.
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Slow query
@ 2014-03-07 06:43 Bikram Kesari Naik <bikram.naik@onmobile.com>
parent: David Johnston <polobo@yahoo.com>
0 siblings, 1 reply; 112+ messages in thread
From: Bikram Kesari Naik @ 2014-03-07 06:43 UTC (permalink / raw)
To: David Johnston <polobo@yahoo.com>; pgsql-performance
Hi David,
We have indexes on all the columns which are used in the where clause and these tables are linked by foreign key constraint.
Thanks,
Bikram
-----Original Message-----
From: pgsql-performance-owner@postgresql.org [mailto:pgsql-performance-owner@postgresql.org] On Behalf Of David Johnston
Sent: Friday, March 07, 2014 11:53 AM
To: pgsql-performance@postgresql.org
Subject: Re: [PERFORM] Slow query
Bikram Kesari Naik wrote
> Hi,
>
> I have a view which joins multiple tables to give me a result. It
> takes more than a minute to give me the result on psql prompt when I
> select all the data from that view.
> The single CPU which is used to run this query is utilized 100%.Even
> if I fire a count(*) it takes 10 Sec. I wanted to know if there is
> anything we can do to speedup this query below 1 sec.
In all likelihood you need to index your foreign keys, and possibly other fields, but as you haven't provided table and index definitions it is hard to say for sure.
Idepeing on how many rows are hidden I'm not sure an unqualified query on this view can run in 1/60th the time even with indexes present - the sequential scans are efficient if the proportion of the tables being returned is high.
David J.
--
View this message in context: http://postgresql.1045698.n5.nabble.com/Slow-query-tp5795077p5795079.html
Sent from the PostgreSQL - performance mailing list archive at Nabble.com.
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
________________________________
DISCLAIMER: The information in this message is confidential and may be legally privileged. It is intended solely for the addressee. Access to this message by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, or distribution of the message, or any action or omission taken by you in reliance on it, is prohibited and may be unlawful. Please immediately contact the sender if you have received this message in error. Further, this e-mail may contain viruses and all reasonable precaution to minimize the risk arising there from is taken by OnMobile. OnMobile is not liable for any damage sustained by you as a result of any virus in this e-mail. All applicable virus checks should be carried out by you before opening this e-mail or any attachment thereto.
Thank you - OnMobile Global Limited.
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Slow query
@ 2014-03-07 07:18 David Johnston <polobo@yahoo.com>
parent: Bikram Kesari Naik <bikram.naik@onmobile.com>
0 siblings, 0 replies; 112+ messages in thread
From: David Johnston @ 2014-03-07 07:18 UTC (permalink / raw)
To: pgsql-performance
Bikram Kesari Naik wrote
> Hi David,
>
> We have indexes on all the columns which are used in the where clause and
> these tables are linked by foreign key constraint.
>
>
> Thanks,
> Bikram
>
> -----Original Message-----
> From:
> pgsql-performance-owner@
> [mailto:
> pgsql-performance-owner@
> ] On Behalf Of David Johnston
> Sent: Friday, March 07, 2014 11:53 AM
> To:
> pgsql-performance@
> Subject: Re: [PERFORM] Slow query
>
> Bikram Kesari Naik wrote
>> Hi,
>>
>> I have a view which joins multiple tables to give me a result. It
>> takes more than a minute to give me the result on psql prompt when I
>> select all the data from that view.
>> The single CPU which is used to run this query is utilized 100%.Even
>> if I fire a count(*) it takes 10 Sec. I wanted to know if there is
>> anything we can do to speedup this query below 1 sec.
>
> In all likelihood you need to index your foreign keys, and possibly other
> fields, but as you haven't provided table and index definitions it is hard
> to say for sure.
>
> Idepeing on how many rows are hidden I'm not sure an unqualified query on
> this view can run in 1/60th the time even with indexes present - the
> sequential scans are efficient if the proportion of the tables being
> returned is high.
>
> David J.
Read these.
https://wiki.postgresql.org/wiki/Using_EXPLAIN
https://wiki.postgresql.org/wiki/SlowQueryQuestions
And note, while the FK thing is likely not relevant in this situation
defining a constraint does not cause an index to be created. Depending on
your usage patterns defining those indexes can be helpful.
One last thought: not only are your row counts high but it seems like your
row sizes may also be large due to them containing binary content. You
likely need to take a different approach to solving whatever unspecified
problem this query is intended to solve if you need sub-second performance.
That all said the main area of improvement for this is system memory
concerns so, as noted in the links above, play with that and see what
happens.
David J.
--
View this message in context: http://postgresql.1045698.n5.nabble.com/Slow-query-tp5795077p5795086.html
Sent from the PostgreSQL - performance mailing list archive at Nabble.com.
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 112+ messages in thread
* Slow query
@ 2014-09-23 12:21 Ross Elliott <ross@wearthefoxhat.com>
0 siblings, 1 reply; 112+ messages in thread
From: Ross Elliott @ 2014-09-23 12:21 UTC (permalink / raw)
To: pgsql-performance
Maybe someone can explain this. The following SQL will reproduce our issue:
DROP TABLE IF EXISTS t1 CASCADE;
CREATE TABLE t1 (name text,
state text);
CREATE INDEX t1_name ON t1(name);
CREATE INDEX t1_state ON t1(state);
CREATE INDEX t1_name_state ON t1(name,state);
-- Create some sample data
DO $$
DECLARE
states text[] := array['UNKNOWN', 'TODO', 'DONE', 'UNKNOWN'];
BEGIN
FOR v IN 1..200000 LOOP
INSERT INTO t1 VALUES('user'||v, states[floor(random()*4)]);
INSERT INTO t1 VALUES('user'||v, states[floor(random()*4)]);
INSERT INTO t1 VALUES('user'||v, states[floor(random()*4)]);
INSERT INTO t1 VALUES('user'||v, states[floor(random()*4)]);
END LOOP;
END $$;
CREATE OR REPLACE FUNCTION state_to_int(state character varying) RETURNS
integer
LANGUAGE plpgsql IMMUTABLE STRICT
AS $$BEGIN
IF state = 'UNKNOWN' THEN RETURN 0;
ELSIF state = 'TODO' THEN RETURN 1;
ELSIF state = 'DONE' THEN RETURN 2;
ELSIF state = 'NOT REQUIRED' THEN RETURN 3;
ELSE RAISE EXCEPTION 'state_to_int called with invalid state value';
END IF;
END;$$;
CREATE OR REPLACE FUNCTION int_to_state(state integer) RETURNS character
varying
LANGUAGE plpgsql IMMUTABLE STRICT
AS $$BEGIN
IF state = 0 THEN RETURN 'UNKNOWN';
ELSIF state = 1 THEN RETURN 'TODO';
ELSIF state = 2 THEN RETURN 'DONE';
ELSIF state = 3 THEN RETURN 'NOT REQUIRED';
ELSE RAISE EXCEPTION 'int_to_state called with invalid state value';
END IF;
END;$$;
-- Why is this a lot slower
explain (analyse, buffers) select name,
int_to_state(min(state_to_int(state))) as status from t1 group by t1.name;
-- Than this?
explain (analyze, buffers) select name, (array['UNKNOWN', 'TODO', 'DONE',
'NOT REQUIRED'])[min(
CASE state
WHEN 'UNKNOWN' THEN 0
WHEN 'TODO' THEN 1
WHEN 'DONE' THEN 2
WHEN 'NOT REQUIRED' THEN 3
END)] AS status from t1 group by t1.name;
-- This is also very much slower
explain (analyze, buffers) select name, (array['UNKNOWN', 'TODO', 'DONE',
'NOT REQUIRED'])[min(state_to_int(state))] AS status from t1 group by
t1.name;
This was done on:
PostgreSQL 9.3.5 on x86_64-unknown-linux-gnu, compiled by gcc (Ubuntu
4.8.2-19ubuntu1) 4.8.2, 64-bit
We get results like this:
QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------------
GroupAggregate (cost=0.42..280042.62 rows=208120 width=15) (actual
time=0.076..2439.066 rows=200000 loops=1)
Buffers: shared hit=53146
-> Index Scan using t1_name on t1 (cost=0.42..21931.42 rows=800000
width=15) (actual time=0.009..229.477 rows=800000 loops=1)
Buffers: shared hit=53146
Total runtime: 2460.860 ms
(5 rows)
QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------------
GroupAggregate (cost=0.42..36012.62 rows=208120 width=15) (actual
time=0.017..559.384 rows=200000 loops=1)
Buffers: shared hit=53146
-> Index Scan using t1_name on t1 (cost=0.42..21931.42 rows=800000
width=15) (actual time=0.008..197.133 rows=800000 loops=1)
Buffers: shared hit=53146
Total runtime: 574.550 ms
(5 rows)
QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------------
GroupAggregate (cost=0.42..228012.62 rows=208120 width=15) (actual
time=0.042..2089.367 rows=200000 loops=1)
Buffers: shared hit=53146
-> Index Scan using t1_name on t1 (cost=0.42..21931.42 rows=800000
width=15) (actual time=0.008..237.854 rows=800000 loops=1)
Buffers: shared hit=53146
Total runtime: 2111.004 ms
(5 rows)
We cannot change our table structure to reflect something more sensible.
What we would really like to know is why using functions is so much slower
than the unreadable method.
Regards
Ross
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Slow query
@ 2014-09-23 13:05 David G Johnston <david.g.johnston@gmail.com>
parent: Ross Elliott <ross@wearthefoxhat.com>
0 siblings, 0 replies; 112+ messages in thread
From: David G Johnston @ 2014-09-23 13:05 UTC (permalink / raw)
To: pgsql-performance
Ross Elliott-2 wrote
> Maybe someone can explain this. The following SQL will reproduce our
> issue:
> DROP TABLE IF EXISTS t1 CASCADE;
> CREATE TABLE t1 (name text,
> state text);
> CREATE INDEX t1_name ON t1(name);
> CREATE INDEX t1_state ON t1(state);
> CREATE INDEX t1_name_state ON t1(name,state);
>
> -- Create some sample data
> DO $$
> DECLARE
> states text[] := array['UNKNOWN', 'TODO', 'DONE', 'UNKNOWN'];
> BEGIN
> FOR v IN 1..200000 LOOP
> INSERT INTO t1 VALUES('user'||v, states[floor(random()*4)]);
> INSERT INTO t1 VALUES('user'||v, states[floor(random()*4)]);
> INSERT INTO t1 VALUES('user'||v, states[floor(random()*4)]);
> INSERT INTO t1 VALUES('user'||v, states[floor(random()*4)]);
> END LOOP;
> END $$;
>
>
> CREATE OR REPLACE FUNCTION state_to_int(state character varying) RETURNS
> integer
> LANGUAGE plpgsql IMMUTABLE STRICT
> AS $$BEGIN
> IF state = 'UNKNOWN' THEN RETURN 0;
> ELSIF state = 'TODO' THEN RETURN 1;
> ELSIF state = 'DONE' THEN RETURN 2;
> ELSIF state = 'NOT REQUIRED' THEN RETURN 3;
> ELSE RAISE EXCEPTION 'state_to_int called with invalid state value';
> END IF;
> END;$$;
>
> CREATE OR REPLACE FUNCTION int_to_state(state integer) RETURNS character
> varying
> LANGUAGE plpgsql IMMUTABLE STRICT
> AS $$BEGIN
> IF state = 0 THEN RETURN 'UNKNOWN';
> ELSIF state = 1 THEN RETURN 'TODO';
> ELSIF state = 2 THEN RETURN 'DONE';
> ELSIF state = 3 THEN RETURN 'NOT REQUIRED';
> ELSE RAISE EXCEPTION 'int_to_state called with invalid state value';
> END IF;
> END;$$;
>
> -- Why is this a lot slower
> explain (analyse, buffers) select name,
> int_to_state(min(state_to_int(state))) as status from t1 group by t1.name;
>
> -- Than this?
> explain (analyze, buffers) select name, (array['UNKNOWN', 'TODO', 'DONE',
> 'NOT REQUIRED'])[min(
> CASE state
> WHEN 'UNKNOWN' THEN 0
> WHEN 'TODO' THEN 1
> WHEN 'DONE' THEN 2
> WHEN 'NOT REQUIRED' THEN 3
> END)] AS status from t1 group by t1.name;
>
> -- This is also very much slower
> explain (analyze, buffers) select name, (array['UNKNOWN', 'TODO', 'DONE',
> 'NOT REQUIRED'])[min(state_to_int(state))] AS status from t1 group by
> t1.name;
>
> This was done on:
> PostgreSQL 9.3.5 on x86_64-unknown-linux-gnu, compiled by gcc (Ubuntu
> 4.8.2-19ubuntu1) 4.8.2, 64-bit
>
> We get results like this:
> QUERY PLAN
>
> -----------------------------------------------------------------------------------------------------------------------------------
> GroupAggregate (cost=0.42..280042.62 rows=208120 width=15) (actual
> time=0.076..2439.066 rows=200000 loops=1)
> Buffers: shared hit=53146
> -> Index Scan using t1_name on t1 (cost=0.42..21931.42 rows=800000
> width=15) (actual time=0.009..229.477 rows=800000 loops=1)
> Buffers: shared hit=53146
> Total runtime: 2460.860 ms
> (5 rows)
>
> QUERY PLAN
>
> -----------------------------------------------------------------------------------------------------------------------------------
> GroupAggregate (cost=0.42..36012.62 rows=208120 width=15) (actual
> time=0.017..559.384 rows=200000 loops=1)
> Buffers: shared hit=53146
> -> Index Scan using t1_name on t1 (cost=0.42..21931.42 rows=800000
> width=15) (actual time=0.008..197.133 rows=800000 loops=1)
> Buffers: shared hit=53146
> Total runtime: 574.550 ms
> (5 rows)
>
> QUERY PLAN
>
> -----------------------------------------------------------------------------------------------------------------------------------
> GroupAggregate (cost=0.42..228012.62 rows=208120 width=15) (actual
> time=0.042..2089.367 rows=200000 loops=1)
> Buffers: shared hit=53146
> -> Index Scan using t1_name on t1 (cost=0.42..21931.42 rows=800000
> width=15) (actual time=0.008..237.854 rows=800000 loops=1)
> Buffers: shared hit=53146
> Total runtime: 2111.004 ms
> (5 rows)
>
>
> We cannot change our table structure to reflect something more sensible.
> What we would really like to know is why using functions is so much slower
> than the unreadable method.
>
> Regards
>
> Ross
Pl/pgsql functions are black boxes and expensive to execute; you should
define these functions as SQL functions and see if that helps.
David J.
--
View this message in context: http://postgresql.1045698.n5.nabble.com/Slow-query-tp5820086p5820096.html
Sent from the PostgreSQL - performance mailing list archive at Nabble.com.
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 112+ messages in thread
* slow query
@ 2015-02-11 06:41 Sathish Nelson <satson.csc17@gmail.com>
0 siblings, 2 replies; 112+ messages in thread
From: Sathish Nelson @ 2015-02-11 06:41 UTC (permalink / raw)
To: pgsql-performance
am connecting three tables in query. one table have 73000 records
another two tables have 138000 records.
but its take 12 sec for show 12402 rows in tables
Tables Structure:
Items Table
CREATE TABLE "C_SAM_Master".items
(
itemno integer NOT NULL,
itemname character varying(250) NOT NULL,
itemcode character varying(250) NOT NULL,
shortname character varying(20) NOT NULL,
aliasname character varying(250) NOT NULL,
aliasnamelanguage character varying(250) NOT NULL,
masteritemno integer NOT NULL,
groupno1 smallint NOT NULL,
groupno2 smallint NOT NULL,
groupno3 smallint NOT NULL,
commodityno smallint NOT NULL,
unitno smallint NOT NULL,
weighttype character(1) NOT NULL,
altunitno smallint NOT NULL,
weight double precision NOT NULL,
reqmrp character(1) NOT NULL,
reqbatch character(1) NOT NULL,
reqmfrdate character(1) NOT NULL,
mfrdateformat character varying(20) NOT NULL,
reqexpdate character(1) NOT NULL,
expdateformat character varying(20) NOT NULL,
expdays1 smallint NOT NULL,
expdays2 character(1) NOT NULL,
expinfodays smallint NOT NULL,
stdsaleratemethod smallint NOT NULL,
salesrateper smallint NOT NULL,
stdprofit1 double precision NOT NULL,
stdprofit2 character(1) NOT NULL,
includestockrep character(1) NOT NULL,
minstock double precision NOT NULL,
minstockunit smallint NOT NULL,
minsaleqtynos double precision NOT NULL,
minsaleqtyunit smallint NOT NULL,
minsaleqty double precision NOT NULL,
description text NOT NULL,
remarks character varying(250) NOT NULL,
actpurchaseorder character(1) NOT NULL,
actpurchase character(1) NOT NULL,
actpurchasereturn character(1) NOT NULL,
actsalesorder character(1) NOT NULL,
actsales character(1) NOT NULL,
actsalesreturn character(1) NOT NULL,
actreceiptnote character(1) NOT NULL,
actdeliverynote character(1) NOT NULL,
actconsumption character(1) NOT NULL,
actproduction character(1) NOT NULL,
actestimate character(1) NOT NULL,
notifypurchaseorder character varying(250) NOT NULL,
notifypurchase character varying(250) NOT NULL,
notifypurchasereturn character varying(250) NOT NULL,
notifysalesorder character varying(250) NOT NULL,
notifysales character varying(250) NOT NULL,
notifysalesreturn character varying(250) NOT NULL,
notifyreceiptnote character varying(250) NOT NULL,
notifydeliverynote character varying(250) NOT NULL,
notifyconsumption character varying(250) NOT NULL,
notifyproduction character varying(250) NOT NULL,
notifyestimate character varying(250) NOT NULL,
act boolean NOT NULL,
recordowner smallint NOT NULL,
lastmodified smallint NOT NULL,
crdate timestamp without time zone NOT NULL,
stdmaxprofit double precision NOT NULL,
commodityname character varying(100) NOT NULL,
lst double precision NOT NULL,
unittype character(1) NOT NULL,
unit1 character varying(15) NOT NULL,
unit2 character varying(15) NOT NULL,
units integer NOT NULL,
unitname character varying(50) NOT NULL,
decimals smallint NOT NULL,
groupname1 character varying(50) NOT NULL,
groupname2 character varying(50) NOT NULL,
groupname3 character varying(50) NOT NULL,
repgroupname character varying(160) NOT NULL,
masteritemname character varying(100) NOT NULL,
altunit1 character varying(15) NOT NULL,
altunit2 character varying(15) NOT NULL,
altunits integer NOT NULL,
altunitname character varying(50) NOT NULL,
altunitdecimals smallint NOT NULL,
CONSTRAINT items_itemno_pk PRIMARY KEY (itemno),
CONSTRAINT items_altunitno_fk FOREIGN KEY (altunitno)
REFERENCES "C_SAM_Master".measureunits (unitno) MATCH SIMPLE
ON UPDATE NO ACTION ON DELETE RESTRICT,
CONSTRAINT items_commodityno_fk FOREIGN KEY (commodityno)
REFERENCES "C_SAM_Master".commodity (commodityno) MATCH SIMPLE
ON UPDATE NO ACTION ON DELETE RESTRICT,
CONSTRAINT items_groupno1_fk FOREIGN KEY (groupno1)
REFERENCES "C_SAM_Master".itemgroup1 (groupno1) MATCH SIMPLE
ON UPDATE NO ACTION ON DELETE RESTRICT,
CONSTRAINT items_groupno2_fk FOREIGN KEY (groupno2)
REFERENCES "C_SAM_Master".itemgroup2 (groupno2) MATCH SIMPLE
ON UPDATE NO ACTION ON DELETE RESTRICT,
CONSTRAINT items_groupno3_fk FOREIGN KEY (groupno3)
REFERENCES "C_SAM_Master".itemgroup3 (groupno3) MATCH SIMPLE
ON UPDATE NO ACTION ON DELETE RESTRICT,
CONSTRAINT items_lastmodified_fk FOREIGN KEY (lastmodified)
REFERENCES appsetup.user1 (userno) MATCH SIMPLE
ON UPDATE NO ACTION ON DELETE RESTRICT,
CONSTRAINT items_masteritemno_fk FOREIGN KEY (masteritemno)
REFERENCES "C_SAM_Master".masteritems (masteritemno) MATCH SIMPLE
ON UPDATE NO ACTION ON DELETE RESTRICT,
CONSTRAINT items_recordowner_fk FOREIGN KEY (recordowner)
REFERENCES appsetup.user1 (userno) MATCH SIMPLE
ON UPDATE NO ACTION ON DELETE RESTRICT,
CONSTRAINT items_unitno_fk FOREIGN KEY (unitno)
REFERENCES "C_SAM_Master".measureunits (unitno) MATCH SIMPLE
ON UPDATE NO ACTION ON DELETE RESTRICT,
CONSTRAINT items_actconsumption_ck CHECK (actconsumption::text <>
''::text),
CONSTRAINT items_actdeliverynote_ck CHECK (actdeliverynote::text <>
''::text),
CONSTRAINT items_actestimate_ck CHECK (actestimate::text <> ''::text),
CONSTRAINT items_actproduction_ck CHECK (actproduction::text <>
''::text),
CONSTRAINT items_actpurchase_ck CHECK (actpurchase::text <> ''::text),
CONSTRAINT items_actpurchaseorder_ck CHECK (actpurchaseorder::text <>
''::text),
CONSTRAINT items_actpurchasereturn_ck CHECK (actpurchasereturn::text <>
''::text),
CONSTRAINT items_actreceiptnote_ck CHECK (actreceiptnote::text <>
''::text),
CONSTRAINT items_actsales_ck CHECK (actsales::text <> ''::text),
CONSTRAINT items_actsalesorder_ck CHECK (actsalesorder::text <>
''::text),
CONSTRAINT items_actsalesreturn_ck CHECK (actsalesreturn::text <>
''::text),
CONSTRAINT items_aliasname_ck CHECK (aliasname::text <> ''::text),
CONSTRAINT items_altunitdecimals_ck CHECK (altunitdecimals >= 0 AND
altunitdecimals <= 3),
CONSTRAINT items_altunits_ck CHECK (altunits >= 0),
CONSTRAINT items_commodityname_ck CHECK (commodityname::text <>
''::text),
CONSTRAINT items_decimals_ck CHECK (decimals >= 0 AND decimals <= 3),
CONSTRAINT items_expdays1_ck CHECK (expdays1 >= 0),
CONSTRAINT items_expinfodays_ck CHECK (expinfodays >= 0),
CONSTRAINT items_includestockrep_ck CHECK (includestockrep::text <>
''::text),
CONSTRAINT items_itemcode_ck CHECK (itemcode::text <> ''::text),
CONSTRAINT items_itemname_ck CHECK (itemname::text <> ''::text),
CONSTRAINT items_itemno_ck CHECK (itemno > 0),
CONSTRAINT items_lst_ck CHECK (lst >= 0::double precision),
CONSTRAINT items_minsaleqty_ck CHECK (minsaleqty >= 0::double precision),
CONSTRAINT items_minsaleqtynos_ck CHECK (minsaleqtynos >= 0::double
precision),
CONSTRAINT items_minsaleqtyunit_ck CHECK (minsaleqtyunit >= 0 AND
minsaleqtyunit <= 2),
CONSTRAINT items_minstock_ck CHECK (minstock >= 0::double precision),
CONSTRAINT items_minstockunit_ck CHECK (minstockunit >= 0 AND minstockunit
<= 2),
CONSTRAINT items_reqbatch_ck CHECK (reqbatch::text <> ''::text),
CONSTRAINT items_reqexpdate_ck CHECK (reqexpdate::text <> ''::text),
CONSTRAINT items_reqmfrdate_ck CHECK (reqmfrdate::text <> ''::text),
CONSTRAINT items_reqmrp_ck CHECK (reqmrp::text <> ''::text),
CONSTRAINT items_salesrateper_ck CHECK (salesrateper >= 0 AND salesrateper
<= 4),
CONSTRAINT items_stdsaleratemethod_ck CHECK (stdsaleratemethod >= 0 AND
stdsaleratemethod <= 2),
CONSTRAINT items_units_ck CHECK (units >= 0),
CONSTRAINT items_unittype_ck CHECK (unittype::text <> ''::text),
CONSTRAINT items_weight_ck CHECK (weight >= 0::double precision),
CONSTRAINT items_weighttype_ck CHECK (weighttype::text <> ''::text)
)
WITH (
OIDS=FALSE
)
TABLESPACE "gpro2_SAM";
ALTER TABLE "C_SAM_Master".items
OWNER TO gpro2user;
-- Index: "C_SAM_Master".items_itemname_uq
-- DROP INDEX "C_SAM_Master".items_itemname_uq;
CREATE UNIQUE INDEX items_itemname_uq
ON "C_SAM_Master".items
USING btree
(lower(itemname::text) COLLATE pg_catalog."default");
-- Rule: rule_del_items ON "C_SAM_Master".items
-- DROP RULE rule_del_items ON "C_SAM_Master".items;
CREATE OR REPLACE RULE rule_del_items AS
ON DELETE TO "C_SAM_Master".items DO ( DELETE FROM
"C_SAM_Master".itembarcode
WHERE itembarcode.itemno = old.itemno;
DELETE FROM "C_SAM_Master".pricelist
WHERE pricelist.itemno = old.itemno;
DELETE FROM "C_SAM_Master".pricelistreview
WHERE pricelistreview.itemno = old.itemno;
);
-- Rule: rule_del_items_c_sam_2014_2015 ON "C_SAM_Master".items
-- DROP RULE rule_del_items_c_sam_2014_2015 ON "C_SAM_Master".items;
CREATE OR REPLACE RULE rule_del_items_c_sam_2014_2015 AS
ON DELETE TO "C_SAM_Master".items DO ( DELETE FROM
"C_SAM_2014-2015".openingstock
WHERE openingstock.itemno = old.itemno;
DELETE FROM "C_SAM_2014-2015".stock
WHERE stock.itemno = old.itemno;
DELETE FROM "C_SAM_2014-2015".packingsetup
WHERE packingsetup.primeitemno = old.itemno;
DELETE FROM "C_SAM_2014-2015".packingsetup
WHERE packingsetup.packingitemno = old.itemno;
DELETE FROM "C_SAM_2014-2015".itemsuppliers
WHERE itemsuppliers.itemno = old.itemno;
DELETE FROM "C_SAM_2014-2015".partyopeningstock
WHERE partyopeningstock.itemno = old.itemno;
DELETE FROM "C_SAM_2014-2015".partystock
WHERE partystock.itemno = old.itemno;
);
Sales 1 Table
CREATE TABLE "C_KA_2014-2015".sales1
(
vtno smallint NOT NULL,
prefix character varying(5) NOT NULL,
idno integer NOT NULL,
suffix character varying(5) NOT NULL,
txno character varying(20) NOT NULL,
txdate timestamp without time zone NOT NULL,
dracno integer NOT NULL,
partyname character varying(100) NOT NULL,
address1 character varying(100) NOT NULL,
address2 character varying(100) NOT NULL,
city character varying(50) NOT NULL,
partytin character varying(30) NOT NULL,
partycstno character varying(30) NOT NULL,
mobileno character varying(15) NOT NULL,
ponos character varying NOT NULL,
pricelevelno smallint NOT NULL,
invno character varying(20) NOT NULL,
duedays smallint NOT NULL,
duedate timestamp without time zone NOT NULL,
paymentmode character varying(10) NOT NULL,
bankrefno character varying(30) NOT NULL,
bankrefdate character varying(10) NOT NULL,
bankfavourname character varying(100) NOT NULL,
bankcrossref character(1) NOT NULL,
bankremarks character varying(100) NOT NULL,
bankdate character varying(10) NOT NULL,
bankstatus character(1) NOT NULL,
bankreconcildate character varying(10) NOT NULL,
stockpointno smallint NOT NULL,
nettotal double precision NOT NULL,
grosswt integer NOT NULL,
tarewt integer NOT NULL,
actualwt double precision NOT NULL,
againstform character varying(15) NOT NULL,
formseriesno character varying(15) NOT NULL,
formno character varying(15) NOT NULL,
formdate character varying(10) NOT NULL,
totalqty double precision NOT NULL,
totalqtyunit character varying(15) NOT NULL,
totalfreeqty double precision NOT NULL,
totalfreeqtyunit character varying(15) NOT NULL,
totalaltqty double precision NOT NULL,
totalaltqtyunit character varying(15) NOT NULL,
orderby smallint NOT NULL,
collectionby smallint NOT NULL,
deliveredby1 character varying(30) NOT NULL,
deliveredby2 character varying(50) NOT NULL,
deliveredrefno character varying(30) NOT NULL,
deliveredrefdate character varying(10) NOT NULL,
goodsdelivered character(1) NOT NULL,
deliveredto1 character varying(50) NOT NULL,
deliveredto2 character varying(50) NOT NULL,
cashrcvd double precision NOT NULL,
remarks character varying(250) NOT NULL,
totalstockvalue double precision NOT NULL,
profit1 double precision NOT NULL,
act boolean NOT NULL,
totalassesvalue double precision NOT NULL,
totaltax double precision NOT NULL,
recordowner smallint NOT NULL,
lastmodified smallint NOT NULL,
crdate timestamp without time zone NOT NULL,
lessadv double precision NOT NULL,
lessadvpartyacno integer NOT NULL,
rateadj double precision NOT NULL,
jobcardtxno character varying(40) NOT NULL,
txtime character varying(8) NOT NULL,
CONSTRAINT sales1_txno_pk PRIMARY KEY (txno),
CONSTRAINT sales1_collectionby_fk FOREIGN KEY (collectionby)
REFERENCES "G_KUMARANGROUPS_Master".employee (empno) MATCH SIMPLE
ON UPDATE NO ACTION ON DELETE RESTRICT,
CONSTRAINT sales1_dracno_fk FOREIGN KEY (dracno)
REFERENCES "C_KA_AcMaster".acledger (acno) MATCH SIMPLE
ON UPDATE NO ACTION ON DELETE RESTRICT,
CONSTRAINT sales1_lastmodified_fk FOREIGN KEY (lastmodified)
REFERENCES appsetup.user1 (userno) MATCH SIMPLE
ON UPDATE NO ACTION ON DELETE RESTRICT,
CONSTRAINT sales1_orderby_fk FOREIGN KEY (orderby)
REFERENCES "G_KUMARANGROUPS_Master".employee (empno) MATCH SIMPLE
ON UPDATE NO ACTION ON DELETE RESTRICT,
CONSTRAINT sales1_pricelevelno_fk FOREIGN KEY (pricelevelno)
REFERENCES "C_KA_AcMaster".acpricelevel (pricelevelno) MATCH SIMPLE
ON UPDATE NO ACTION ON DELETE RESTRICT,
CONSTRAINT sales1_recordowner_fk FOREIGN KEY (recordowner)
REFERENCES appsetup.user1 (userno) MATCH SIMPLE
ON UPDATE NO ACTION ON DELETE RESTRICT,
CONSTRAINT sales1_vto_fk FOREIGN KEY (vtno)
REFERENCES "C_KA_2014-2015".acvouchertype (vtno) MATCH SIMPLE
ON UPDATE NO ACTION ON DELETE RESTRICT,
CONSTRAINT sales1_vtnoprefixidnosuffix_uq UNIQUE (vtno, prefix, idno,
suffix),
CONSTRAINT sales1_duedays_ck CHECK (duedays >= 0),
CONSTRAINT sales1_idno_ck CHECK (idno > 0),
CONSTRAINT sales1_lessadv_ck CHECK (lessadv >= 0::double precision),
CONSTRAINT sales1_lessadvpartyacno_ck CHECK (lessadvpartyacno >= 0),
CONSTRAINT sales1_partyname_ck CHECK (partyname::text <> ''::text),
CONSTRAINT sales1_paymentmode_ck CHECK (paymentmode::text <> ''::text),
CONSTRAINT sales1_stockpointno_ck CHECK (stockpointno >= 0)
)
WITH (
OIDS=FALSE
)
TABLESPACE "gpro2_KA";
ALTER TABLE "C_KA_2014-2015".sales1
OWNER TO gpro2user;
-- Index: "C_KA_2014-2015".sales1_acno
-- DROP INDEX "C_KA_2014-2015".sales1_acno;
CREATE INDEX sales1_acno
ON "C_KA_2014-2015".sales1
USING btree
(dracno);
-- Index: "C_KA_2014-2015".sales1_txdate
-- DROP INDEX "C_KA_2014-2015".sales1_txdate;
CREATE INDEX sales1_txdate
ON "C_KA_2014-2015".sales1
USING btree
(txdate);
-- Rule: rule_del_sales ON "C_KA_2014-2015".sales1
-- DROP RULE rule_del_sales ON "C_KA_2014-2015".sales1;
CREATE OR REPLACE RULE rule_del_sales AS
ON DELETE TO "C_KA_2014-2015".sales1 DO ( DELETE FROM
"C_KA_2014-2015".packingitemsautopost
WHERE packingitemsautopost.transtype::text = 'Sales'::text AND
packingitemsautopost.txno::text = old.txno::text;
DELETE FROM "C_KA_2014-2015".sales6
WHERE sales6.txno::text = old.txno::text;
DELETE FROM "C_KA_2014-2015".sales5
WHERE sales5.txno::text = old.txno::text;
DELETE FROM "C_KA_2014-2015".sales4
WHERE sales4.txno::text = old.txno::text;
DELETE FROM "C_KA_2014-2015".sales3
WHERE sales3.txno::text = old.txno::text;
DELETE FROM "C_KA_2014-2015".sales2
WHERE sales2.txno::text = old.txno::text;
);
-- Trigger: trg_sales1 on "C_KA_2014-2015".sales1
-- DROP TRIGGER trg_sales1 ON "C_KA_2014-2015".sales1;
CREATE TRIGGER trg_sales1
AFTER UPDATE OF act
ON "C_KA_2014-2015".sales1
FOR EACH ROW
EXECUTE PROCEDURE fn_trg_sales('sales1');
-- Trigger: trg_sales1acpost on "C_KA_2014-2015".sales1
-- DROP TRIGGER trg_sales1acpost ON "C_KA_2014-2015".sales1;
CREATE TRIGGER trg_sales1acpost
AFTER INSERT OR UPDATE OF txdate OR DELETE
ON "C_KA_2014-2015".sales1
FOR EACH ROW
EXECUTE PROCEDURE fn_trg_sales1acpost();
Sales 2 Table
CREATE TABLE "C_KA_2014-2015".sales2
(
txno character varying(20) NOT NULL,
slno smallint NOT NULL,
itemno integer NOT NULL,
rowkey smallint NOT NULL,
mrp double precision NOT NULL,
batchno character varying(20) NOT NULL,
expdate character varying(10) NOT NULL,
qty1 double precision NOT NULL,
qty2 double precision NOT NULL,
freeqty1 double precision NOT NULL,
freeqty2 double precision NOT NULL,
altqty1 double precision NOT NULL,
altqty2 double precision NOT NULL,
rate double precision NOT NULL,
rateper smallint NOT NULL,
basedvalue double precision NOT NULL,
tradedis1 double precision NOT NULL,
tradedis2 double precision NOT NULL,
totaltradis double precision NOT NULL,
adnldis1 double precision NOT NULL,
adnldis2 double precision NOT NULL,
totaladnldis double precision NOT NULL,
adnlcostbeforevat double precision NOT NULL,
assesvalue double precision NOT NULL,
cst1 double precision NOT NULL,
cst2 double precision NOT NULL,
lst1 double precision NOT NULL,
lst2 double precision NOT NULL,
amount double precision NOT NULL,
itemdescription text NOT NULL,
adnlcostafterevat double precision NOT NULL,
nsr double precision NOT NULL,
totalqty double precision NOT NULL,
totalfreeqty double precision NOT NULL,
totalaltqty double precision NOT NULL,
primaryacno integer NOT NULL,
taxacno integer NOT NULL,
itemstockvalue double precision NOT NULL,
itemprofit1 double precision NOT NULL,
cliamscheme character(1) NOT NULL,
netrate double precision NOT NULL,
pricelistrate double precision NOT NULL,
CONSTRAINT sales2_txnoslno_pk PRIMARY KEY (txno, slno),
CONSTRAINT sales2_itemno_fk FOREIGN KEY (itemno)
REFERENCES "G_KUMARANGROUPS_Master".items (itemno) MATCH SIMPLE
ON UPDATE NO ACTION ON DELETE RESTRICT,
CONSTRAINT sales2_txno_fk FOREIGN KEY (txno)
REFERENCES "C_KA_2014-2015".sales1 (txno) MATCH SIMPLE
ON UPDATE NO ACTION ON DELETE RESTRICT,
CONSTRAINT sales2_rowkey_uq UNIQUE (rowkey, txno),
CONSTRAINT sales2_cst1_ck CHECK (cst1 >= 0::double precision),
CONSTRAINT sales2_lst1_ck CHECK (lst1 >= 0::double precision),
CONSTRAINT sales2_mrp_ck CHECK (mrp >= 0::double precision),
CONSTRAINT sales2_netrate_ck CHECK (netrate >= 0::double precision),
CONSTRAINT sales2_nsr_ck CHECK (nsr >= 0::double precision),
CONSTRAINT sales2_pricelistrate_ck CHECK (pricelistrate >= 0::double
precision),
CONSTRAINT sales2_primaryacno_ck CHECK (primaryacno >= 0),
CONSTRAINT sales2_rate_ck CHECK (rate >= 0::double precision),
CONSTRAINT sales2_rateper_ck CHECK (rateper >= 0 AND rateper <= 4),
CONSTRAINT sales2_rowkey_ck CHECK (rowkey > 0),
CONSTRAINT sales2_slno_ck CHECK (slno > 0),
CONSTRAINT sales2_taxacno_ck CHECK (taxacno >= 0),
CONSTRAINT sales2_totalfreeqty_ck CHECK ((totalqty + totalfreeqty) <>
0::double precision),
CONSTRAINT sales2_totalqty_ck CHECK ((totalqty + totalfreeqty) <>
0::double precision)
)
WITH (
OIDS=FALSE
)
TABLESPACE "gpro2_KA";
ALTER TABLE "C_KA_2014-2015".sales2
OWNER TO gpro2user;
-- Index: "C_KA_2014-2015".sales2_itemno
-- DROP INDEX "C_KA_2014-2015".sales2_itemno;
CREATE INDEX sales2_itemno
ON "C_KA_2014-2015".sales2
USING btree
(itemno);
-- Index: "C_KA_2014-2015".sales2_txno
-- DROP INDEX "C_KA_2014-2015".sales2_txno;
CREATE INDEX sales2_txno
ON "C_KA_2014-2015".sales2
USING btree
(txno COLLATE pg_catalog."default");
-- Trigger: trg_sales2 on "C_KA_2014-2015".sales2
-- DROP TRIGGER trg_sales2 ON "C_KA_2014-2015".sales2;
CREATE TRIGGER trg_sales2
AFTER INSERT OR DELETE
ON "C_KA_2014-2015".sales2
FOR EACH ROW
EXECUTE PROCEDURE fn_trg_sales2();
Query:
select grp,disp,alisdisp,ord,'' as adnlorder,'' as calcorder,sum(case when
ord =3 then qty end) as qty,sum(case when ord=3 then freeqty end) as
freeqty,max(case when ord=3 then unit1 end) as unit1,sum(altqty) as
altqty,max(altunit1) as altunit1,sum(discount) as discount,sum(amount) as
amount,sum(itemprofit) as itemprofit,0.00 as profitper,sum(itemstockvalue)
as itemstockvalue from (select
unnest(array[repgroupname,repgroupname||'-'||masteritemname,repgroupname||'-'||masteritemname||'-'||itemname])
as grp,unnest(array[case when repgroupname ='' then 'UnGrouped' else
repgroupname end,masteritemname,itemname]) as
disp,unnest(array['','',aliasnamelanguage]) as alisdisp,unnest(array[1,2,3])
as ord,cast(case when units > 1 then cast(case when sum(qty) > 0 then
floor(sum(qty)/units) else ceil(sum(qty)/units) end as text) when
(mod(cast(sum(qty) as integer),units))<>0 then '.' ||
abs(cast(mod(cast(sum(qty) as integer),units) as integer)) else
cast(sum(qty) as text) end as double precision) as qty,cast(case when units
> 1 then cast(case when sum(freeqty) > 0 then floor(sum(freeqty)/units) else
ceil(sum(freeqty)/units) end as text) when (mod(cast(sum(freeqty) as
integer),units))<>0 then '.' || abs(cast(mod(cast(sum(freeqty) as
integer),units) as integer)) else cast(sum(freeqty) as text) end as double
precision) as freeqty,unit1,cast(case when altunits > 1 then cast(case when
sum(altqty) > 0 then floor(sum(altqty)/altunits) else
ceil(sum(altqty)/altunits) end as text) when (mod(cast(sum(altqty) as
integer),altunits))<>0 then '.' || abs(cast(mod(cast(sum(altqty) as
integer),altunits) as integer)) else cast(sum(altqty) as text) end as double
precision) as altqty,altunit1,sum(discount) as discount,sum(itemstockvalue)
as itemstockvalue,sum(itemprofit) as itemprofit,sum(amount) as
amount,shortname from (select
i.repgroupname,i.aliasnamelanguage,i.masteritemname,i.itemname,i.groupname1,i.groupname2,i.groupname3,i.units,i.unit1,i.unit2,i.altunit1,i.altunit2,i.altunits,sum(s2.totalqty)
as qty,sum(s2.totalfreeqty) as freeqty,sum(s2.totalaltqty) as
altqty,sum(s2.totaltradis + totaladnldis) as discount,sum(itemstockvalue) as
itemstockvalue,sum(itemprofit1) as itemprofit,sum(s2.amount) as
amount,'KA'::text as shortname from "C_KA_2014-2015".sales1 s1 inner join
"C_KA_2014-2015".sales2 s2 on s1.txno=s2.txno inner join
"G_KUMARANGROUPS_Master".items i on i.itemno=s2.itemno where s1.act='t' and
s1.txdate >= '01/04/2014' and s1.txdate <= '30/01/2015'group by
i.repgroupname,i.aliasnamelanguage,i.groupname1,i.groupname2,i.groupname3,i.units,i.unit1,i.unit2,i.altunit1,i.altunit2,i.altunits,i.itemname,i.masteritemname
) as tt group by
grp,disp,alisdisp,units,altunits,ord,shortname,unit1,altunit1 order by
grp,disp ) as tab where disp <> ''
group by grp,disp,alisdisp,ord order by grp,disp
Explain Analysis and Buffers
"GroupAggregate (cost=3586024.69..3617755.12 rows=72944 width=160) (actual
time=11819.837..11884.868 rows=12064 loops=1)"
" Buffers: shared hit=4462 read=9825, temp read=6381 written=6361"
" -> Sort (cost=3586024.69..3587848.28 rows=729435 width=160) (actual
time=11819.780..11831.894 rows=12068 loops=1)"
" Sort Key: tab.grp, tab.disp, tab.alisdisp, tab.ord"
" Sort Method: external sort Disk: 1336kB"
" Buffers: shared hit=4462 read=9825, temp read=6381 written=6361"
" -> Subquery Scan on tab (cost=2742202.68..3342958.78 rows=729435
width=160) (actual time=11424.007..11727.170 rows=12068 loops=1)"
" Filter: ((tab.disp)::text <> ''::text)"
" Rows Removed by Filter: 7"
" Buffers: shared hit=4462 read=9825, temp read=6214
written=6194"
" -> GroupAggregate (cost=2742202.68..3333795.03 rows=733100
width=115) (actual time=11424.001..11703.904 rows=12075 loops=1)"
" Buffers: shared hit=4462 read=9825, temp read=6214
written=6194"
" -> Sort (cost=2742202.68..2760528.43 rows=7330300
width=115) (actual time=11423.951..11543.478 rows=36183 loops=1)"
" Sort Key: (unnest(ARRAY[tt.repgroupname,
((((tt.repgroupname)::text || '-'::text) ||
(tt.masteritemname)::text))::character varying, ((((((tt.repgroupname)::text
|| '-'::text) || (tt.masteritemname)::text) || '-'::text) || (tt.ite (...)"
" Sort Method: external merge Disk: 3552kB"
" Buffers: shared hit=4462 read=9825, temp
read=6214 written=6194"
" -> Subquery Scan on tt
(cost=56047.61..102407.06 rows=7330300 width=115) (actual
time=8877.785..11023.746 rows=36183 loops=1)"
" Buffers: shared hit=4462 read=9825, temp
read=5768 written=5748"
" -> GroupAggregate
(cost=56047.61..63373.22 rows=73303 width=96) (actual
time=8877.762..10906.503 rows=12061 loops=1)"
" Buffers: shared hit=4462 read=9825,
temp read=5768 written=5748"
" -> Sort (cost=56047.61..56347.27
rows=119865 width=96) (actual time=8877.576..10555.267 rows=119714
loops=1)"
" Sort Key: i.repgroupname,
i.aliasnamelanguage, i.groupname1, i.groupname2, i.groupname3, i.units,
i.unit1, i.unit2, i.altunit1, i.altunit2, i.altunits, i.itemname,
i.masteritemname"
" Sort Method: external merge
Disk: 12432kB"
" Buffers: shared hit=4462
read=9825, temp read=5768 written=5748"
" -> Hash Join
(cost=13948.80..33644.37 rows=119865 width=96) (actual
time=617.917..1756.039 rows=119714 loops=1)"
" Hash Cond: (s2.itemno =
i.itemno)"
" Buffers: shared hit=4462
read=9825, temp read=3098 written=3078"
" -> Hash Join
(cost=8849.48..23064.41 rows=119865 width=68) (actual time=339.948..1054.380
rows=119714 loops=1)"
" Hash Cond:
((s2.txno)::text = (s1.txno)::text)"
" Buffers: shared
hit=1585 read=9825, temp read=1539 written=1533"
" -> Seq Scan on
sales2 s2 (cost=0.00..7490.64 rows=144964 width=76) (actual
time=0.023..262.043 rows=144964 loops=1)"
" Buffers:
shared hit=814 read=5227"
" -> Hash
(cost=7196.35..7196.35 rows=100731 width=8) (actual time=339.873..339.873
rows=100850 loops=1)"
" Buckets: 4096
Batches: 4 Memory Usage: 803kB"
" Buffers:
shared hit=771 read=4598, temp written=257"
" -> Seq Scan
on sales1 s1 (cost=0.00..7196.35 rows=100731 width=8) (actual
time=0.029..230.250 rows=100850 loops=1)"
" Filter:
(act AND (txdate >= '2014-04-01 00:00:00'::timestamp without time zone) AND
(txdate <= '2015-01-30 00:00:00'::timestamp without time zone))"
" Rows
Removed by Filter: 20973"
"
Buffers: shared hit=771 read=4598"
" -> Hash
(cost=3610.03..3610.03 rows=73303 width=36) (actual time=277.327..277.327
rows=73303 loops=1)"
" Buckets: 2048
Batches: 8 Memory Usage: 593kB"
" Buffers: shared
hit=2877, temp written=475"
" -> Seq Scan on
items i (cost=0.00..3610.03 rows=73303 width=36) (actual
time=0.007..153.900 rows=73303 loops=1)"
" Buffers:
shared hit=2877"
"Total runtime: 11897.250 ms"
My Hardware is
InterCore
CPU 3.10 CHZ
2.89 GB RAM
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: slow query
@ 2015-02-11 13:33 Torsten Förtsch <torsten.foertsch@gmx.net>
parent: Sathish Nelson <satson.csc17@gmail.com>
1 sibling, 0 replies; 112+ messages in thread
From: Torsten Förtsch @ 2015-02-11 13:33 UTC (permalink / raw)
To: Sathish Nelson <satson.csc17@gmail.com>; pgsql-performance
On 11/02/15 07:41, Sathish Nelson wrote:
> am connecting three tables in query. one table have 73000 records
>
> another two tables have 138000 records.
>
> but its take 12 sec for show 12402 rows in tables
you need more work_mem. The query plan shows multiple external sort nodes.
Otherwise, the query and the plan I received here are too badly
formatted for me to comprehend.
Torsten
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: slow query
@ 2015-02-11 13:44 Tom Lane <tgl@sss.pgh.pa.us>
parent: Sathish Nelson <satson.csc17@gmail.com>
1 sibling, 0 replies; 112+ messages in thread
From: Tom Lane @ 2015-02-11 13:44 UTC (permalink / raw)
To: Sathish Nelson <satson.csc17@gmail.com>; +Cc: pgsql-performance
Sathish Nelson <satson.csc17@gmail.com> writes:
> am connecting three tables in query. one table have 73000 records
> another two tables have 138000 records.
> but its take 12 sec for show 12402 rows in tables
Increasing work_mem would make those sort steps faster ...
regards, tom lane
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 112+ messages in thread
* Slow Query
@ 2015-08-12 02:34 robbyc <robcampbell73@gmail.com>
0 siblings, 2 replies; 112+ messages in thread
From: robbyc @ 2015-08-12 02:34 UTC (permalink / raw)
To: pgsql-performance
Hi,
I am new to optimizing queries and i'm getting a slow running time
(~1.5secs) with the following SQL:
SELECT distinct("Vacancy"."ID"), "Vacancy"."JobTitle",
"Vacancy"."DateCreated", "Vacancy"."CustomAccess"
, "Department"."Name" as "Department", list("Occupation"."Name") as
"Occupation", "Vacancy"."PositionNo"
, "Vacancy"."Template" from
"Vacancy"
LEFT JOIN "CategoryOption_TableRow" as "c_22"
ON ("c_22"."RowID" = "Vacancy"."ID"
and "c_22"."Category_TableID" = 22)
LEFT JOIN "CategoryOption" as "Occupation"
ON ("Occupation"."ID" = "c_22"."CategoryOptionID")
LEFT JOIN "TableRow_TableRow" as "t_33"
ON ("t_33"."Table1RowID" = "Vacancy"."ID"
and "t_33"."Table_TableID" = 33 )
LEFT JOIN "Department"
ON ("Department"."ID" = "t_33"."Table2RowID" and
"Department"."Active" = 't' and "Department"
."ClientID" = 263)
JOIN "CategoryOption_TableRow" as "c_50" ON ("c_50"."Category_TableID"= 50
and "c_50"."RowID" = "Vacancy"
."ID" and "c_50"."CategoryOptionID"=19205)
WHERE "Vacancy"."ClientID" = 263 AND "Vacancy"."ID" NOT IN(SELECT
DISTINCT("Vacancy"."ID")
FROM "Vacancy" join "CategoryOption_TableRow" "ct126" on
("ct126"."Category_TableID" = 126
and "RowID" = "Vacancy"."ID")
left join "Workflow" on ("Workflow"."VacancyID" = "Vacancy"."ID"
and "Workflow"."Level"
= 1)
left join "CategoryOption_TableRow" "c30" on ("c30"."RowID" =
"Workflow"."ID" and "c30"."Category_TableID"
= 30 and "c30"."CategoryOptionID" = 21923)
WHERE "Template" AND "ct126"."CategoryOptionID"
IN(34024,35254,35255,35256)) and "Vacancy"
."Template" = 't'
GROUP BY "Vacancy"."ID", "Vacancy"."JobTitle", "Vacancy"."DateCreated",
"Vacancy"."CustomAccess", "Department"
."Name", "Vacancy"."PositionNo", "Vacancy"."Template"
UNION
SELECT distinct("Vacancy"."ID"), "Vacancy"."JobTitle",
"Vacancy"."DateCreated", "Vacancy"."CustomAccess"
, "Department"."Name" as "Department", list("Occupation"."Name") as
"Occupation", "Vacancy"."PositionNo"
, "Vacancy"."Template" from
"Vacancy"
LEFT JOIN "CategoryOption_TableRow" as "c_22"
ON ("c_22"."RowID" = "Vacancy"."ID"
and "c_22"."Category_TableID" = 22)
LEFT JOIN "CategoryOption" as "Occupation"
ON ("Occupation"."ID" = "c_22"."CategoryOptionID")
LEFT JOIN "TableRow_TableRow" as "t_33"
ON ("t_33"."Table1RowID" = "Vacancy"."ID"
and "t_33"."Table_TableID" = 33 )
LEFT JOIN "Department"
ON ("Department"."ID" = "t_33"."Table2RowID" and
"Department"."Active" = 't' and "Department"
."ClientID" = 263)
JOIN "CategoryOption_TableRow" as "c_50" ON ("c_50"."Category_TableID"= 50
and "c_50"."RowID" = "Vacancy"
."ID" and "c_50"."CategoryOptionID"=19205)
WHERE "Vacancy"."ClientID" = 263 AND "Vacancy"."ID" NOT IN(SELECT
DISTINCT("Vacancy"."ID")
FROM "Vacancy" join "CategoryOption_TableRow" "ct126" on
("ct126"."Category_TableID" = 126
and "RowID" = "Vacancy"."ID")
left join "Workflow" on ("Workflow"."VacancyID" = "Vacancy"."ID"
and "Workflow"."Level"
= 1)
left join "CategoryOption_TableRow" "c30" on ("c30"."RowID" =
"Workflow"."ID" and "c30"."Category_TableID"
= 30 and "c30"."CategoryOptionID" = 21923)
WHERE "Template" AND "ct126"."CategoryOptionID"
IN(34024,35254,35255,35256)) and "Vacancy"
."Template" <> 't' AND "Vacancy"."Level" = 1
GROUP BY "Vacancy"."ID", "Vacancy"."JobTitle", "Vacancy"."DateCreated",
"Vacancy"."CustomAccess", "Department"
."Name", "Vacancy"."PositionNo", "Vacancy"."Template"
ORDER BY "JobTitle"
Running explain analyze gives me the following information:
http://explain.depesz.com/s/pdC <http://explain.depesz.com/s/pdC;
For a total runtime: 2877.157 ms
If i remove the left joins on Department and TableRow_TableRow this reduces
the run time by about a third.
Additionally removing CategoryOption and CategoryOption_TableRow joins
further reduces by a about a third.
Given that i need both these joins for the information retrieved by them,
what would be the best way to re-factor this query so it runs faster?
Looking at the output of explain analyze the hash aggregates and sort seem
to be the primary issue.
Thanks in advance
--
View this message in context: http://postgresql.nabble.com/Slow-Query-tp5861835.html
Sent from the PostgreSQL - performance mailing list archive at Nabble.com.
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Slow Query
@ 2015-08-12 04:08 Venkata Balaji N <nag1010@gmail.com>
parent: robbyc <robcampbell73@gmail.com>
1 sibling, 1 reply; 112+ messages in thread
From: Venkata Balaji N @ 2015-08-12 04:08 UTC (permalink / raw)
To: robbyc <robcampbell73@gmail.com>; +Cc: pgsql-performance
On Wed, Aug 12, 2015 at 12:34 PM, robbyc <robcampbell73@gmail.com> wrote:
> Hi,
>
> I am new to optimizing queries and i'm getting a slow running time
> (~1.5secs) with the following SQL:
>
> SELECT distinct("Vacancy"."ID"), "Vacancy"."JobTitle",
> "Vacancy"."DateCreated", "Vacancy"."CustomAccess"
> , "Department"."Name" as "Department", list("Occupation"."Name") as
> "Occupation", "Vacancy"."PositionNo"
> , "Vacancy"."Template" from
> "Vacancy"
> LEFT JOIN "CategoryOption_TableRow" as "c_22"
> ON ("c_22"."RowID" = "Vacancy"."ID"
> and "c_22"."Category_TableID" = 22)
> LEFT JOIN "CategoryOption" as "Occupation"
> ON ("Occupation"."ID" = "c_22"."CategoryOptionID")
> LEFT JOIN "TableRow_TableRow" as "t_33"
> ON ("t_33"."Table1RowID" = "Vacancy"."ID"
> and "t_33"."Table_TableID" = 33 )
> LEFT JOIN "Department"
> ON ("Department"."ID" = "t_33"."Table2RowID" and
> "Department"."Active" = 't' and "Department"
> ."ClientID" = 263)
> JOIN "CategoryOption_TableRow" as "c_50" ON ("c_50"."Category_TableID"= 50
> and "c_50"."RowID" = "Vacancy"
> ."ID" and "c_50"."CategoryOptionID"=19205)
> WHERE "Vacancy"."ClientID" = 263 AND "Vacancy"."ID" NOT IN(SELECT
> DISTINCT("Vacancy"."ID")
> FROM "Vacancy" join "CategoryOption_TableRow" "ct126" on
> ("ct126"."Category_TableID" = 126
> and "RowID" = "Vacancy"."ID")
> left join "Workflow" on ("Workflow"."VacancyID" =
> "Vacancy"."ID"
> and "Workflow"."Level"
> = 1)
> left join "CategoryOption_TableRow" "c30" on ("c30"."RowID" =
> "Workflow"."ID" and "c30"."Category_TableID"
> = 30 and "c30"."CategoryOptionID" = 21923)
> WHERE "Template" AND "ct126"."CategoryOptionID"
> IN(34024,35254,35255,35256)) and "Vacancy"
> ."Template" = 't'
> GROUP BY "Vacancy"."ID", "Vacancy"."JobTitle", "Vacancy"."DateCreated",
> "Vacancy"."CustomAccess", "Department"
> ."Name", "Vacancy"."PositionNo", "Vacancy"."Template"
> UNION
> SELECT distinct("Vacancy"."ID"), "Vacancy"."JobTitle",
> "Vacancy"."DateCreated", "Vacancy"."CustomAccess"
> , "Department"."Name" as "Department", list("Occupation"."Name") as
> "Occupation", "Vacancy"."PositionNo"
> , "Vacancy"."Template" from
> "Vacancy"
> LEFT JOIN "CategoryOption_TableRow" as "c_22"
> ON ("c_22"."RowID" = "Vacancy"."ID"
> and "c_22"."Category_TableID" = 22)
> LEFT JOIN "CategoryOption" as "Occupation"
> ON ("Occupation"."ID" = "c_22"."CategoryOptionID")
> LEFT JOIN "TableRow_TableRow" as "t_33"
> ON ("t_33"."Table1RowID" = "Vacancy"."ID"
> and "t_33"."Table_TableID" = 33 )
> LEFT JOIN "Department"
> ON ("Department"."ID" = "t_33"."Table2RowID" and
> "Department"."Active" = 't' and "Department"
> ."ClientID" = 263)
> JOIN "CategoryOption_TableRow" as "c_50" ON ("c_50"."Category_TableID"= 50
> and "c_50"."RowID" = "Vacancy"
> ."ID" and "c_50"."CategoryOptionID"=19205)
> WHERE "Vacancy"."ClientID" = 263 AND "Vacancy"."ID" NOT IN(SELECT
> DISTINCT("Vacancy"."ID")
> FROM "Vacancy" join "CategoryOption_TableRow" "ct126" on
> ("ct126"."Category_TableID" = 126
> and "RowID" = "Vacancy"."ID")
> left join "Workflow" on ("Workflow"."VacancyID" =
> "Vacancy"."ID"
> and "Workflow"."Level"
> = 1)
> left join "CategoryOption_TableRow" "c30" on ("c30"."RowID" =
> "Workflow"."ID" and "c30"."Category_TableID"
> = 30 and "c30"."CategoryOptionID" = 21923)
> WHERE "Template" AND "ct126"."CategoryOptionID"
> IN(34024,35254,35255,35256)) and "Vacancy"
> ."Template" <> 't' AND "Vacancy"."Level" = 1
> GROUP BY "Vacancy"."ID", "Vacancy"."JobTitle", "Vacancy"."DateCreated",
> "Vacancy"."CustomAccess", "Department"
> ."Name", "Vacancy"."PositionNo", "Vacancy"."Template"
> ORDER BY "JobTitle"
>
> Running explain analyze gives me the following information:
> http://explain.depesz.com/s/pdC <http://explain.depesz.com/s/pdC;
> For a total runtime: 2877.157 ms
>
> If i remove the left joins on Department and TableRow_TableRow this reduces
> the run time by about a third.
> Additionally removing CategoryOption and CategoryOption_TableRow joins
> further reduces by a about a third.
>
> Given that i need both these joins for the information retrieved by them,
> what would be the best way to re-factor this query so it runs faster?
>
> Looking at the output of explain analyze the hash aggregates and sort seem
> to be the primary issue.
>
The query has got a distinct and group-by/order-by clauses which seems to
be taking time. Without looking at much details of the query code and Table
size etc, did you try increasing the work_mem and then execute the query
and see if that helps ? This will reduce the on-disk IO for sorting. Also,
Vacancy.JobTitle seems to be a non-index column.
Regards,
Venkata Balaji
Fujitsu Australia
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Slow Query
@ 2015-08-12 05:29 robbyc <robcampbell73@gmail.com>
parent: Venkata Balaji N <nag1010@gmail.com>
0 siblings, 1 reply; 112+ messages in thread
From: robbyc @ 2015-08-12 05:29 UTC (permalink / raw)
To: pgsql-performance
Hi Venkata,
work_mem was set to 72MB, increased to 144MB, no change.
Added an index of type varchar_pattern_ops to Vacancy.JobTitle, this did
not help either.
On Wed, Aug 12, 2015 at 2:09 PM, Venkata Balaji N [via PostgreSQL] <
ml-node+s1045698n5861839h42@n5.nabble.com> wrote:
> On Wed, Aug 12, 2015 at 12:34 PM, robbyc <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=5861839&i=0>; wrote:
>
>> Hi,
>>
>> I am new to optimizing queries and i'm getting a slow running time
>> (~1.5secs) with the following SQL:
>>
>> SELECT distinct("Vacancy"."ID"), "Vacancy"."JobTitle",
>> "Vacancy"."DateCreated", "Vacancy"."CustomAccess"
>> , "Department"."Name" as "Department", list("Occupation"."Name") as
>> "Occupation", "Vacancy"."PositionNo"
>> , "Vacancy"."Template" from
>> "Vacancy"
>> LEFT JOIN "CategoryOption_TableRow" as "c_22"
>> ON ("c_22"."RowID" = "Vacancy"."ID"
>> and "c_22"."Category_TableID" = 22)
>> LEFT JOIN "CategoryOption" as "Occupation"
>> ON ("Occupation"."ID" = "c_22"."CategoryOptionID")
>> LEFT JOIN "TableRow_TableRow" as "t_33"
>> ON ("t_33"."Table1RowID" = "Vacancy"."ID"
>> and "t_33"."Table_TableID" = 33 )
>> LEFT JOIN "Department"
>> ON ("Department"."ID" = "t_33"."Table2RowID" and
>> "Department"."Active" = 't' and "Department"
>> ."ClientID" = 263)
>> JOIN "CategoryOption_TableRow" as "c_50" ON ("c_50"."Category_TableID"= 50
>> and "c_50"."RowID" = "Vacancy"
>> ."ID" and "c_50"."CategoryOptionID"=19205)
>> WHERE "Vacancy"."ClientID" = 263 AND "Vacancy"."ID" NOT IN(SELECT
>> DISTINCT("Vacancy"."ID")
>> FROM "Vacancy" join "CategoryOption_TableRow" "ct126" on
>> ("ct126"."Category_TableID" = 126
>> and "RowID" = "Vacancy"."ID")
>> left join "Workflow" on ("Workflow"."VacancyID" =
>> "Vacancy"."ID"
>> and "Workflow"."Level"
>> = 1)
>> left join "CategoryOption_TableRow" "c30" on ("c30"."RowID" =
>> "Workflow"."ID" and "c30"."Category_TableID"
>> = 30 and "c30"."CategoryOptionID" = 21923)
>> WHERE "Template" AND "ct126"."CategoryOptionID"
>> IN(34024,35254,35255,35256)) and "Vacancy"
>> ."Template" = 't'
>> GROUP BY "Vacancy"."ID", "Vacancy"."JobTitle", "Vacancy"."DateCreated",
>> "Vacancy"."CustomAccess", "Department"
>> ."Name", "Vacancy"."PositionNo", "Vacancy"."Template"
>> UNION
>> SELECT distinct("Vacancy"."ID"), "Vacancy"."JobTitle",
>> "Vacancy"."DateCreated", "Vacancy"."CustomAccess"
>> , "Department"."Name" as "Department", list("Occupation"."Name") as
>> "Occupation", "Vacancy"."PositionNo"
>> , "Vacancy"."Template" from
>> "Vacancy"
>> LEFT JOIN "CategoryOption_TableRow" as "c_22"
>> ON ("c_22"."RowID" = "Vacancy"."ID"
>> and "c_22"."Category_TableID" = 22)
>> LEFT JOIN "CategoryOption" as "Occupation"
>> ON ("Occupation"."ID" = "c_22"."CategoryOptionID")
>> LEFT JOIN "TableRow_TableRow" as "t_33"
>> ON ("t_33"."Table1RowID" = "Vacancy"."ID"
>> and "t_33"."Table_TableID" = 33 )
>> LEFT JOIN "Department"
>> ON ("Department"."ID" = "t_33"."Table2RowID" and
>> "Department"."Active" = 't' and "Department"
>> ."ClientID" = 263)
>> JOIN "CategoryOption_TableRow" as "c_50" ON ("c_50"."Category_TableID"= 50
>> and "c_50"."RowID" = "Vacancy"
>> ."ID" and "c_50"."CategoryOptionID"=19205)
>> WHERE "Vacancy"."ClientID" = 263 AND "Vacancy"."ID" NOT IN(SELECT
>> DISTINCT("Vacancy"."ID")
>> FROM "Vacancy" join "CategoryOption_TableRow" "ct126" on
>> ("ct126"."Category_TableID" = 126
>> and "RowID" = "Vacancy"."ID")
>> left join "Workflow" on ("Workflow"."VacancyID" =
>> "Vacancy"."ID"
>> and "Workflow"."Level"
>> = 1)
>> left join "CategoryOption_TableRow" "c30" on ("c30"."RowID" =
>> "Workflow"."ID" and "c30"."Category_TableID"
>> = 30 and "c30"."CategoryOptionID" = 21923)
>> WHERE "Template" AND "ct126"."CategoryOptionID"
>> IN(34024,35254,35255,35256)) and "Vacancy"
>> ."Template" <> 't' AND "Vacancy"."Level" = 1
>> GROUP BY "Vacancy"."ID", "Vacancy"."JobTitle", "Vacancy"."DateCreated",
>> "Vacancy"."CustomAccess", "Department"
>> ."Name", "Vacancy"."PositionNo", "Vacancy"."Template"
>> ORDER BY "JobTitle"
>>
>> Running explain analyze gives me the following information:
>> http://explain.depesz.com/s/pdC <http://explain.depesz.com/s/pdC;
>
>
>> For a total runtime: 2877.157 ms
>>
>> If i remove the left joins on Department and TableRow_TableRow this
>> reduces
>> the run time by about a third.
>> Additionally removing CategoryOption and CategoryOption_TableRow joins
>> further reduces by a about a third.
>>
>> Given that i need both these joins for the information retrieved by them,
>> what would be the best way to re-factor this query so it runs faster?
>>
>> Looking at the output of explain analyze the hash aggregates and sort seem
>> to be the primary issue.
>>
>
> The query has got a distinct and group-by/order-by clauses which seems to
> be taking time. Without looking at much details of the query code and Table
> size etc, did you try increasing the work_mem and then execute the query
> and see if that helps ? This will reduce the on-disk IO for sorting. Also,
> Vacancy.JobTitle seems to be a non-index column.
>
> Regards,
> Venkata Balaji
>
> Fujitsu Australia
>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://postgresql.nabble.com/Slow-Query-tp5861835p5861839.html
> To unsubscribe from Slow Query, click here
> <http://postgresql.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=5861835&...;
> .
> NAML
> <http://postgresql.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabbl...;
>
--
Regards
Robert Campbell
+61412062971
robcampbell73@gmail.com
--
View this message in context: http://postgresql.nabble.com/Slow-Query-tp5861835p5861850.html
Sent from the PostgreSQL - performance mailing list archive at Nabble.com.
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Slow Query
@ 2015-08-12 06:08 Venkata Balaji N <nag1010@gmail.com>
parent: robbyc <robcampbell73@gmail.com>
0 siblings, 0 replies; 112+ messages in thread
From: Venkata Balaji N @ 2015-08-12 06:08 UTC (permalink / raw)
To: robbyc <robcampbell73@gmail.com>; +Cc: pgsql-performance
On Wed, Aug 12, 2015 at 3:29 PM, robbyc <robcampbell73@gmail.com> wrote:
> Hi Venkata,
>
> work_mem was set to 72MB, increased to 144MB, no change.
>
Increasing work_mem depends on various other factors like Table size
(amount of data being sorted), available memory etc.
> Added an index of type varchar_pattern_ops to Vacancy.JobTitle, this did
> not help either.
>
Sorry, I did not mean to say that an Index must be added straight away. The
column must be eligible to have an Index. Meaning, Index will be
beneficial if created on a column with high number of distinct values.
If either of the above does not help, then options to rewrite the query
must be explored.
Thanks,
Venkata Balaji N
Fujitsu Australia
>
> On Wed, Aug 12, 2015 at 2:09 PM, Venkata Balaji N [via PostgreSQL] <[hidden
> email] <http:///user/SendEmail.jtp?type=node&node=5861850&i=0>; wrote:
>
>> On Wed, Aug 12, 2015 at 12:34 PM, robbyc <[hidden email]
>> <http:///user/SendEmail.jtp?type=node&node=5861839&i=0>; wrote:
>>
>>> Hi,
>>>
>>> I am new to optimizing queries and i'm getting a slow running time
>>> (~1.5secs) with the following SQL:
>>>
>>> SELECT distinct("Vacancy"."ID"), "Vacancy"."JobTitle",
>>> "Vacancy"."DateCreated", "Vacancy"."CustomAccess"
>>> , "Department"."Name" as "Department", list("Occupation"."Name") as
>>> "Occupation", "Vacancy"."PositionNo"
>>> , "Vacancy"."Template" from
>>> "Vacancy"
>>> LEFT JOIN "CategoryOption_TableRow" as "c_22"
>>> ON ("c_22"."RowID" = "Vacancy"."ID"
>>> and "c_22"."Category_TableID" = 22)
>>> LEFT JOIN "CategoryOption" as "Occupation"
>>> ON ("Occupation"."ID" = "c_22"."CategoryOptionID")
>>> LEFT JOIN "TableRow_TableRow" as "t_33"
>>> ON ("t_33"."Table1RowID" = "Vacancy"."ID"
>>> and "t_33"."Table_TableID" = 33 )
>>> LEFT JOIN "Department"
>>> ON ("Department"."ID" = "t_33"."Table2RowID" and
>>> "Department"."Active" = 't' and "Department"
>>> ."ClientID" = 263)
>>> JOIN "CategoryOption_TableRow" as "c_50" ON ("c_50"."Category_TableID"=
>>> 50
>>> and "c_50"."RowID" = "Vacancy"
>>> ."ID" and "c_50"."CategoryOptionID"=19205)
>>> WHERE "Vacancy"."ClientID" = 263 AND "Vacancy"."ID" NOT IN(SELECT
>>> DISTINCT("Vacancy"."ID")
>>> FROM "Vacancy" join "CategoryOption_TableRow" "ct126" on
>>> ("ct126"."Category_TableID" = 126
>>> and "RowID" = "Vacancy"."ID")
>>> left join "Workflow" on ("Workflow"."VacancyID" =
>>> "Vacancy"."ID"
>>> and "Workflow"."Level"
>>> = 1)
>>> left join "CategoryOption_TableRow" "c30" on ("c30"."RowID" =
>>> "Workflow"."ID" and "c30"."Category_TableID"
>>> = 30 and "c30"."CategoryOptionID" = 21923)
>>> WHERE "Template" AND "ct126"."CategoryOptionID"
>>> IN(34024,35254,35255,35256)) and "Vacancy"
>>> ."Template" = 't'
>>> GROUP BY "Vacancy"."ID", "Vacancy"."JobTitle", "Vacancy"."DateCreated",
>>> "Vacancy"."CustomAccess", "Department"
>>> ."Name", "Vacancy"."PositionNo", "Vacancy"."Template"
>>> UNION
>>> SELECT distinct("Vacancy"."ID"), "Vacancy"."JobTitle",
>>> "Vacancy"."DateCreated", "Vacancy"."CustomAccess"
>>> , "Department"."Name" as "Department", list("Occupation"."Name") as
>>> "Occupation", "Vacancy"."PositionNo"
>>> , "Vacancy"."Template" from
>>> "Vacancy"
>>> LEFT JOIN "CategoryOption_TableRow" as "c_22"
>>> ON ("c_22"."RowID" = "Vacancy"."ID"
>>> and "c_22"."Category_TableID" = 22)
>>> LEFT JOIN "CategoryOption" as "Occupation"
>>> ON ("Occupation"."ID" = "c_22"."CategoryOptionID")
>>> LEFT JOIN "TableRow_TableRow" as "t_33"
>>> ON ("t_33"."Table1RowID" = "Vacancy"."ID"
>>> and "t_33"."Table_TableID" = 33 )
>>> LEFT JOIN "Department"
>>> ON ("Department"."ID" = "t_33"."Table2RowID" and
>>> "Department"."Active" = 't' and "Department"
>>> ."ClientID" = 263)
>>> JOIN "CategoryOption_TableRow" as "c_50" ON ("c_50"."Category_TableID"=
>>> 50
>>> and "c_50"."RowID" = "Vacancy"
>>> ."ID" and "c_50"."CategoryOptionID"=19205)
>>> WHERE "Vacancy"."ClientID" = 263 AND "Vacancy"."ID" NOT IN(SELECT
>>> DISTINCT("Vacancy"."ID")
>>> FROM "Vacancy" join "CategoryOption_TableRow" "ct126" on
>>> ("ct126"."Category_TableID" = 126
>>> and "RowID" = "Vacancy"."ID")
>>> left join "Workflow" on ("Workflow"."VacancyID" =
>>> "Vacancy"."ID"
>>> and "Workflow"."Level"
>>> = 1)
>>> left join "CategoryOption_TableRow" "c30" on ("c30"."RowID" =
>>> "Workflow"."ID" and "c30"."Category_TableID"
>>> = 30 and "c30"."CategoryOptionID" = 21923)
>>> WHERE "Template" AND "ct126"."CategoryOptionID"
>>> IN(34024,35254,35255,35256)) and "Vacancy"
>>> ."Template" <> 't' AND "Vacancy"."Level" = 1
>>> GROUP BY "Vacancy"."ID", "Vacancy"."JobTitle", "Vacancy"."DateCreated",
>>> "Vacancy"."CustomAccess", "Department"
>>> ."Name", "Vacancy"."PositionNo", "Vacancy"."Template"
>>> ORDER BY "JobTitle"
>>>
>>> Running explain analyze gives me the following information:
>>> http://explain.depesz.com/s/pdC <http://explain.depesz.com/s/pdC;
>>
>>
>>> For a total runtime: 2877.157 ms
>>>
>>> If i remove the left joins on Department and TableRow_TableRow this
>>> reduces
>>> the run time by about a third.
>>> Additionally removing CategoryOption and CategoryOption_TableRow joins
>>> further reduces by a about a third.
>>>
>>> Given that i need both these joins for the information retrieved by them,
>>> what would be the best way to re-factor this query so it runs faster?
>>>
>>> Looking at the output of explain analyze the hash aggregates and sort
>>> seem
>>> to be the primary issue.
>>>
>>
>> The query has got a distinct and group-by/order-by clauses which seems to
>> be taking time. Without looking at much details of the query code and Table
>> size etc, did you try increasing the work_mem and then execute the query
>> and see if that helps ? This will reduce the on-disk IO for sorting. Also,
>> Vacancy.JobTitle seems to be a non-index column.
>>
>> Regards,
>> Venkata Balaji
>>
>> Fujitsu Australia
>>
>>
>>
>> ------------------------------
>> If you reply to this email, your message will be added to the discussion
>> below:
>> http://postgresql.nabble.com/Slow-Query-tp5861835p5861839.html
>> To unsubscribe from Slow Query, click here.
>> NAML
>> <http://postgresql.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabbl...;
>>
>
>
>
> --
> Regards
>
> Robert Campbell
> +61412062971
> [hidden email] <http:///user/SendEmail.jtp?type=node&node=5861850&i=1;
>
> ------------------------------
> View this message in context: Re: Slow Query
> <http://postgresql.nabble.com/Slow-Query-tp5861835p5861850.html;
>
> Sent from the PostgreSQL - performance mailing list archive
> <http://postgresql.nabble.com/PostgreSQL-performance-f2050081.html; at
> Nabble.com.
>
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Slow Query
@ 2015-08-12 11:34 Vik Fearing <vik@2ndquadrant.fr>
parent: robbyc <robcampbell73@gmail.com>
1 sibling, 1 reply; 112+ messages in thread
From: Vik Fearing @ 2015-08-12 11:34 UTC (permalink / raw)
To: pgsql-performance; robcampbell73@gmail.com
On 08/12/2015 04:34 AM, robbyc wrote:
> Hi,
>
> I am new to optimizing queries and i'm getting a slow running time
> (~1.5secs) with the following SQL:
Before mucking about with work_mem and indexes, the first thing to do is
rewrite this query correctly. Here are just some of the things wrong
with the query as written:
* You're doing a DISTINCT on the same set of columns also in a GROUP BY.
This is redundant and causes needless deduplication.
* You're joining two GROUPed BY then DISTINCTed queries using the UNION
operator which will do yet another pass for deduplication.
* You've got the entire query repeated for just a simple difference in
the global WHERE clause. These can be merged.
* You've kept LEFT JOINs in the subquery but you don't use any values
from them. These can be safely removed altogether.
* You're using a NOT IN clause which is almost never what you want. Use
NOT EXISTS instead.
What is this list() function? How is it defined? Can it be replaced
with string_agg()?
You're not doing yourself any favors at all with all this quoting and
mixed case stuff.
Here is a rewritten version, please let me know how it performs:
SELECT "Vacancy"."ID",
"Vacancy"."JobTitle",
"Vacancy"."DateCreated",
"Vacancy"."CustomAccess",
"Department"."Name" as "Department",
list("Occupation"."Name") as "Occupation",
"Vacancy"."PositionNo",
"Vacancy"."Template"
FROM "Vacancy"
JOIN "CategoryOption_TableRow" as "c_50" ON (
"c_50"."Category_TableID"= 50
AND "c_50"."RowID" = "Vacancy"."ID"
AND "c_50"."CategoryOptionID"=19205)
LEFT JOIN "CategoryOption_TableRow" as "c_22" ON (
"c_22"."RowID" = "Vacancy"."ID"
AND "c_22"."Category_TableID" = 22)
LEFT JOIN "CategoryOption" as "Occupation" ON (
"Occupation"."ID" = "c_22"."CategoryOptionID")
LEFT JOIN "TableRow_TableRow" as "t_33" ON (
"t_33"."Table1RowID" = "Vacancy"."ID"
AND "t_33"."Table_TableID" = 33)
LEFT JOIN "Department" ON (
"Department"."ID" = "t_33"."Table2RowID"
AND "Department"."Active" = 't'
AND "Department"."ClientID" = 263)
WHERE "Vacancy"."ClientID" = 263
AND NOT EXISTS (
SELECT 1
FROM "Vacancy" as _Vacancy
JOIN "CategoryOption_TableRow" "ct126" on (
"ct126"."Category_TableID" = 126
AND "RowID" = _Vacancy."ID")
WHERE _Vacancy."Template"
AND "ct126"."CategoryOptionID" IN (34024,35254,35255,35256)
AND _Vacancy."ID" = "Vacancy"."ID")
AND ("Vacancy"."Template" = 't' OR "Vacancy"."Level" = 1)
GROUP BY "Vacancy"."ID", "Vacancy"."JobTitle", "Vacancy"."DateCreated",
"Vacancy"."CustomAccess", "Department"."Name",
"Vacancy"."PositionNo", "Vacancy"."Template"
> SELECT distinct("Vacancy"."ID"), "Vacancy"."JobTitle",
> "Vacancy"."DateCreated", "Vacancy"."CustomAccess"
> , "Department"."Name" as "Department", list("Occupation"."Name") as
> "Occupation", "Vacancy"."PositionNo"
> , "Vacancy"."Template" from
> "Vacancy"
> LEFT JOIN "CategoryOption_TableRow" as "c_22"
> ON ("c_22"."RowID" = "Vacancy"."ID"
> and "c_22"."Category_TableID" = 22)
> LEFT JOIN "CategoryOption" as "Occupation"
> ON ("Occupation"."ID" = "c_22"."CategoryOptionID")
> LEFT JOIN "TableRow_TableRow" as "t_33"
> ON ("t_33"."Table1RowID" = "Vacancy"."ID"
> and "t_33"."Table_TableID" = 33 )
> LEFT JOIN "Department"
> ON ("Department"."ID" = "t_33"."Table2RowID" and
> "Department"."Active" = 't' and "Department"
> ."ClientID" = 263)
> JOIN "CategoryOption_TableRow" as "c_50" ON ("c_50"."Category_TableID"= 50
> and "c_50"."RowID" = "Vacancy"
> ."ID" and "c_50"."CategoryOptionID"=19205)
> WHERE "Vacancy"."ClientID" = 263 AND "Vacancy"."ID" NOT IN(SELECT
> DISTINCT("Vacancy"."ID")
> FROM "Vacancy" join "CategoryOption_TableRow" "ct126" on
> ("ct126"."Category_TableID" = 126
> and "RowID" = "Vacancy"."ID")
> left join "Workflow" on ("Workflow"."VacancyID" = "Vacancy"."ID"
> and "Workflow"."Level"
> = 1)
> left join "CategoryOption_TableRow" "c30" on ("c30"."RowID" =
> "Workflow"."ID" and "c30"."Category_TableID"
> = 30 and "c30"."CategoryOptionID" = 21923)
> WHERE "Template" AND "ct126"."CategoryOptionID"
> IN(34024,35254,35255,35256)) and "Vacancy"
> ."Template" = 't'
> GROUP BY "Vacancy"."ID", "Vacancy"."JobTitle", "Vacancy"."DateCreated",
> "Vacancy"."CustomAccess", "Department"
> ."Name", "Vacancy"."PositionNo", "Vacancy"."Template"
> UNION
> SELECT distinct("Vacancy"."ID"), "Vacancy"."JobTitle",
> "Vacancy"."DateCreated", "Vacancy"."CustomAccess"
> , "Department"."Name" as "Department", list("Occupation"."Name") as
> "Occupation", "Vacancy"."PositionNo"
> , "Vacancy"."Template" from
> "Vacancy"
> LEFT JOIN "CategoryOption_TableRow" as "c_22"
> ON ("c_22"."RowID" = "Vacancy"."ID"
> and "c_22"."Category_TableID" = 22)
> LEFT JOIN "CategoryOption" as "Occupation"
> ON ("Occupation"."ID" = "c_22"."CategoryOptionID")
> LEFT JOIN "TableRow_TableRow" as "t_33"
> ON ("t_33"."Table1RowID" = "Vacancy"."ID"
> and "t_33"."Table_TableID" = 33 )
> LEFT JOIN "Department"
> ON ("Department"."ID" = "t_33"."Table2RowID" and
> "Department"."Active" = 't' and "Department"
> ."ClientID" = 263)
> JOIN "CategoryOption_TableRow" as "c_50" ON ("c_50"."Category_TableID"= 50
> and "c_50"."RowID" = "Vacancy"
> ."ID" and "c_50"."CategoryOptionID"=19205)
> WHERE "Vacancy"."ClientID" = 263 AND "Vacancy"."ID" NOT IN(SELECT
> DISTINCT("Vacancy"."ID")
> FROM "Vacancy" join "CategoryOption_TableRow" "ct126" on
> ("ct126"."Category_TableID" = 126
> and "RowID" = "Vacancy"."ID")
> left join "Workflow" on ("Workflow"."VacancyID" = "Vacancy"."ID"
> and "Workflow"."Level"
> = 1)
> left join "CategoryOption_TableRow" "c30" on ("c30"."RowID" =
> "Workflow"."ID" and "c30"."Category_TableID"
> = 30 and "c30"."CategoryOptionID" = 21923)
> WHERE "Template" AND "ct126"."CategoryOptionID"
> IN(34024,35254,35255,35256)) and "Vacancy"
> ."Template" <> 't' AND "Vacancy"."Level" = 1
> GROUP BY "Vacancy"."ID", "Vacancy"."JobTitle", "Vacancy"."DateCreated",
> "Vacancy"."CustomAccess", "Department"
> ."Name", "Vacancy"."PositionNo", "Vacancy"."Template"
> ORDER BY "JobTitle"
>
> Running explain analyze gives me the following information:
> http://explain.depesz.com/s/pdC <http://explain.depesz.com/s/pdC;
>
> For a total runtime: 2877.157 ms
>
> If i remove the left joins on Department and TableRow_TableRow this reduces
> the run time by about a third.
> Additionally removing CategoryOption and CategoryOption_TableRow joins
> further reduces by a about a third.
>
> Given that i need both these joins for the information retrieved by them,
> what would be the best way to re-factor this query so it runs faster?
>
> Looking at the output of explain analyze the hash aggregates and sort seem
> to be the primary issue.
>
> Thanks in advance
--
Vik Fearing +33 6 46 75 15 36
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Slow Query
@ 2015-08-13 00:50 robbyc <robcampbell73@gmail.com>
parent: Vik Fearing <vik@2ndquadrant.fr>
0 siblings, 1 reply; 112+ messages in thread
From: robbyc @ 2015-08-13 00:50 UTC (permalink / raw)
To: pgsql-performance
Hi Vik,
Thanks for your feedback, very helpful.
I modified your query slightly, this will return all vacancy templates and
all level 1 vacancies which arent templates, and does so in about
~800-900ms less, an great improvement on the original query.
SELECT "Vacancy"."ID",
"Vacancy"."JobTitle",
"Vacancy"."DateCreated",
"Vacancy"."CustomAccess",
"Department"."Name" as "Department",
list("Occupation"."Name") as "Occupation",
"Vacancy"."PositionNo",
"Vacancy"."Template"
FROM "Vacancy"
JOIN "CategoryOption_TableRow" as "c_50" ON (
"c_50"."Category_TableID"= 50
AND "c_50"."RowID" = "Vacancy"."ID"
AND "c_50"."CategoryOptionID"=19205)
LEFT JOIN "CategoryOption_TableRow" as "c_22" ON (
"c_22"."RowID" = "Vacancy"."ID"
AND "c_22"."Category_TableID" = 22)
LEFT JOIN "CategoryOption" as "Occupation" ON (
"Occupation"."ID" = "c_22"."CategoryOptionID")
LEFT JOIN "TableRow_TableRow" as "t_33" ON (
"t_33"."Table1RowID" = "Vacancy"."ID"
AND "t_33"."Table_TableID" = 33)
LEFT JOIN "Department" ON (
"Department"."ID" = "t_33"."Table2RowID"
AND "Department"."Active" = 't'
AND "Department"."ClientID" = 263)
WHERE "Vacancy"."ClientID" = 263
AND NOT EXISTS (
SELECT 1
FROM "Vacancy" as "v"
JOIN "CategoryOption_TableRow" "ct126" on (
"ct126"."Category_TableID" = 126
AND "RowID" = "v"."ID")
WHERE "v"."Template"
AND "ct126"."CategoryOptionID" IN (34024,35254,35255,35256)
AND "v"."ID" = "Vacancy"."ID")
AND ("Vacancy"."Template" OR ("Vacancy"."Template" = 'f' AND
"Vacancy"."Level" = 1))
GROUP BY "Vacancy"."ID", "Vacancy"."JobTitle", "Vacancy"."DateCreated",
"Vacancy"."CustomAccess", "Department"."Name",
"Vacancy"."PositionNo", "Vacancy"."Template"
On Wed, Aug 12, 2015 at 9:35 PM, Vik Fearing-3 [via PostgreSQL] <
ml-node+s1045698n5861873h96@n5.nabble.com> wrote:
> On 08/12/2015 04:34 AM, robbyc wrote:
> > Hi,
> >
> > I am new to optimizing queries and i'm getting a slow running time
> > (~1.5secs) with the following SQL:
>
> Before mucking about with work_mem and indexes, the first thing to do is
> rewrite this query correctly. Here are just some of the things wrong
> with the query as written:
>
> * You're doing a DISTINCT on the same set of columns also in a GROUP BY.
> This is redundant and causes needless deduplication.
>
> * You're joining two GROUPed BY then DISTINCTed queries using the UNION
> operator which will do yet another pass for deduplication.
>
> * You've got the entire query repeated for just a simple difference in
> the global WHERE clause. These can be merged.
>
> * You've kept LEFT JOINs in the subquery but you don't use any values
> from them. These can be safely removed altogether.
>
> * You're using a NOT IN clause which is almost never what you want. Use
> NOT EXISTS instead.
>
> What is this list() function? How is it defined? Can it be replaced
> with string_agg()?
>
> You're not doing yourself any favors at all with all this quoting and
> mixed case stuff.
>
> Here is a rewritten version, please let me know how it performs:
>
> SELECT "Vacancy"."ID",
> "Vacancy"."JobTitle",
> "Vacancy"."DateCreated",
> "Vacancy"."CustomAccess",
> "Department"."Name" as "Department",
> list("Occupation"."Name") as "Occupation",
> "Vacancy"."PositionNo",
> "Vacancy"."Template"
> FROM "Vacancy"
> JOIN "CategoryOption_TableRow" as "c_50" ON (
> "c_50"."Category_TableID"= 50
> AND "c_50"."RowID" = "Vacancy"."ID"
> AND "c_50"."CategoryOptionID"=19205)
> LEFT JOIN "CategoryOption_TableRow" as "c_22" ON (
> "c_22"."RowID" = "Vacancy"."ID"
> AND "c_22"."Category_TableID" = 22)
> LEFT JOIN "CategoryOption" as "Occupation" ON (
> "Occupation"."ID" = "c_22"."CategoryOptionID")
> LEFT JOIN "TableRow_TableRow" as "t_33" ON (
> "t_33"."Table1RowID" = "Vacancy"."ID"
> AND "t_33"."Table_TableID" = 33)
> LEFT JOIN "Department" ON (
> "Department"."ID" = "t_33"."Table2RowID"
> AND "Department"."Active" = 't'
> AND "Department"."ClientID" = 263)
> WHERE "Vacancy"."ClientID" = 263
> AND NOT EXISTS (
> SELECT 1
> FROM "Vacancy" as _Vacancy
> JOIN "CategoryOption_TableRow" "ct126" on (
> "ct126"."Category_TableID" = 126
> AND "RowID" = _Vacancy."ID")
> WHERE _Vacancy."Template"
> AND "ct126"."CategoryOptionID" IN (34024,35254,35255,35256)
> AND _Vacancy."ID" = "Vacancy"."ID")
> AND ("Vacancy"."Template" = 't' OR "Vacancy"."Level" = 1)
> GROUP BY "Vacancy"."ID", "Vacancy"."JobTitle", "Vacancy"."DateCreated",
> "Vacancy"."CustomAccess", "Department"."Name",
> "Vacancy"."PositionNo", "Vacancy"."Template"
>
>
> > SELECT distinct("Vacancy"."ID"), "Vacancy"."JobTitle",
> > "Vacancy"."DateCreated", "Vacancy"."CustomAccess"
> > , "Department"."Name" as "Department", list("Occupation"."Name") as
> > "Occupation", "Vacancy"."PositionNo"
> > , "Vacancy"."Template" from
> > "Vacancy"
> > LEFT JOIN "CategoryOption_TableRow" as "c_22"
> > ON ("c_22"."RowID" = "Vacancy"."ID"
> > and "c_22"."Category_TableID" = 22)
> > LEFT JOIN "CategoryOption" as "Occupation"
> > ON ("Occupation"."ID" = "c_22"."CategoryOptionID")
> > LEFT JOIN "TableRow_TableRow" as "t_33"
> > ON ("t_33"."Table1RowID" = "Vacancy"."ID"
> > and "t_33"."Table_TableID" = 33 )
> > LEFT JOIN "Department"
> > ON ("Department"."ID" = "t_33"."Table2RowID" and
> > "Department"."Active" = 't' and "Department"
> > ."ClientID" = 263)
> > JOIN "CategoryOption_TableRow" as "c_50" ON ("c_50"."Category_TableID"=
> 50
> > and "c_50"."RowID" = "Vacancy"
> > ."ID" and "c_50"."CategoryOptionID"=19205)
> > WHERE "Vacancy"."ClientID" = 263 AND "Vacancy"."ID" NOT IN(SELECT
> > DISTINCT("Vacancy"."ID")
> > FROM "Vacancy" join "CategoryOption_TableRow" "ct126" on
> > ("ct126"."Category_TableID" = 126
> > and "RowID" = "Vacancy"."ID")
> > left join "Workflow" on ("Workflow"."VacancyID" =
> "Vacancy"."ID"
> > and "Workflow"."Level"
> > = 1)
> > left join "CategoryOption_TableRow" "c30" on ("c30"."RowID"
> =
> > "Workflow"."ID" and "c30"."Category_TableID"
> > = 30 and "c30"."CategoryOptionID" = 21923)
> > WHERE "Template" AND "ct126"."CategoryOptionID"
> > IN(34024,35254,35255,35256)) and "Vacancy"
> > ."Template" = 't'
> > GROUP BY "Vacancy"."ID", "Vacancy"."JobTitle", "Vacancy"."DateCreated",
> > "Vacancy"."CustomAccess", "Department"
> > ."Name", "Vacancy"."PositionNo", "Vacancy"."Template"
> > UNION
> > SELECT distinct("Vacancy"."ID"), "Vacancy"."JobTitle",
> > "Vacancy"."DateCreated", "Vacancy"."CustomAccess"
> > , "Department"."Name" as "Department", list("Occupation"."Name") as
> > "Occupation", "Vacancy"."PositionNo"
> > , "Vacancy"."Template" from
> > "Vacancy"
> > LEFT JOIN "CategoryOption_TableRow" as "c_22"
> > ON ("c_22"."RowID" = "Vacancy"."ID"
> > and "c_22"."Category_TableID" = 22)
> > LEFT JOIN "CategoryOption" as "Occupation"
> > ON ("Occupation"."ID" = "c_22"."CategoryOptionID")
> > LEFT JOIN "TableRow_TableRow" as "t_33"
> > ON ("t_33"."Table1RowID" = "Vacancy"."ID"
> > and "t_33"."Table_TableID" = 33 )
> > LEFT JOIN "Department"
> > ON ("Department"."ID" = "t_33"."Table2RowID" and
> > "Department"."Active" = 't' and "Department"
> > ."ClientID" = 263)
> > JOIN "CategoryOption_TableRow" as "c_50" ON ("c_50"."Category_TableID"=
> 50
> > and "c_50"."RowID" = "Vacancy"
> > ."ID" and "c_50"."CategoryOptionID"=19205)
> > WHERE "Vacancy"."ClientID" = 263 AND "Vacancy"."ID" NOT IN(SELECT
> > DISTINCT("Vacancy"."ID")
> > FROM "Vacancy" join "CategoryOption_TableRow" "ct126" on
> > ("ct126"."Category_TableID" = 126
> > and "RowID" = "Vacancy"."ID")
> > left join "Workflow" on ("Workflow"."VacancyID" =
> "Vacancy"."ID"
> > and "Workflow"."Level"
> > = 1)
> > left join "CategoryOption_TableRow" "c30" on ("c30"."RowID"
> =
> > "Workflow"."ID" and "c30"."Category_TableID"
> > = 30 and "c30"."CategoryOptionID" = 21923)
> > WHERE "Template" AND "ct126"."CategoryOptionID"
> > IN(34024,35254,35255,35256)) and "Vacancy"
> > ."Template" <> 't' AND "Vacancy"."Level" = 1
> > GROUP BY "Vacancy"."ID", "Vacancy"."JobTitle", "Vacancy"."DateCreated",
> > "Vacancy"."CustomAccess", "Department"
> > ."Name", "Vacancy"."PositionNo", "Vacancy"."Template"
> > ORDER BY "JobTitle"
> >
> > Running explain analyze gives me the following information:
> > http://explain.depesz.com/s/pdC <http://explain.depesz.com/s/pdC;
> >
> > For a total runtime: 2877.157 ms
> >
> > If i remove the left joins on Department and TableRow_TableRow this
> reduces
> > the run time by about a third.
> > Additionally removing CategoryOption and CategoryOption_TableRow joins
> > further reduces by a about a third.
> >
> > Given that i need both these joins for the information retrieved by
> them,
> > what would be the best way to re-factor this query so it runs faster?
> >
> > Looking at the output of explain analyze the hash aggregates and sort
> seem
> > to be the primary issue.
> >
> > Thanks in advance
>
>
> --
> Vik Fearing +33 6 46 75 15 36
> http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support
>
>
> --
> Sent via pgsql-performance mailing list ([hidden email]
> <http:///user/SendEmail.jtp?type=node&node=5861873&i=0;)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-performance
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://postgresql.nabble.com/Slow-Query-tp5861835p5861873.html
> To unsubscribe from Slow Query, click here
> <http://postgresql.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=5861835&...;
> .
> NAML
> <http://postgresql.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabbl...;
>
--
Regards
Robert Campbell
+61412062971
robcampbell73@gmail.com
--
View this message in context: http://postgresql.nabble.com/Slow-Query-tp5861835p5861961.html
Sent from the PostgreSQL - performance mailing list archive at Nabble.com.
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Slow Query
@ 2015-08-13 09:21 林士博 <lin@repica.co.jp>
parent: robbyc <robcampbell73@gmail.com>
0 siblings, 1 reply; 112+ messages in thread
From: 林士博 @ 2015-08-13 09:21 UTC (permalink / raw)
To: robbyc <robcampbell73@gmail.com>; +Cc: pgsql-performance
In the 'not exists' cluster, you do not have to search table "Vacancy as v"
again.
I think it would be faster to use the outer Vacancy table as below.
In your case, that do the same work.
NOT EXISTS (
SELECT 1
FROM "CategoryOption_TableRow" "ct126"
WHERE "Vacancy"."Template"
AND "ct126"."CategoryOptionID" IN (34024,35254,35255,35256)
AND "ct126"."Category_TableID" = 126
AND "ct126"."RowID" = "Vacancy"."ID"
)
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Slow Query
@ 2015-08-14 03:15 robbyc <robcampbell73@gmail.com>
parent: 林士博 <lin@repica.co.jp>
0 siblings, 1 reply; 112+ messages in thread
From: robbyc @ 2015-08-14 03:15 UTC (permalink / raw)
To: pgsql-performance
Hi,
Doing this returns 0 records
On Thu, Aug 13, 2015 at 7:22 PM, 林士博 [via PostgreSQL] <
ml-node+s1045698n5862008h81@n5.nabble.com> wrote:
> In the 'not exists' cluster, you do not have to search table "Vacancy as
> v" again.
> I think it would be faster to use the outer Vacancy table as below.
> In your case, that do the same work.
>
> NOT EXISTS (
> SELECT 1
> FROM "CategoryOption_TableRow" "ct126"
> WHERE "Vacancy"."Template"
> AND "ct126"."CategoryOptionID" IN (34024,35254,35255,35256)
> AND "ct126"."Category_TableID" = 126
> AND "ct126"."RowID" = "Vacancy"."ID"
> )
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://postgresql.nabble.com/Slow-Query-tp5861835p5862008.html
> To unsubscribe from Slow Query, click here
> <http://postgresql.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=5861835&...;
> .
> NAML
> <http://postgresql.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabbl...;
>
--
Regards
Robert Campbell
+61412062971
robcampbell73@gmail.com
--
View this message in context: http://postgresql.nabble.com/Slow-Query-tp5861835p5862122.html
Sent from the PostgreSQL - performance mailing list archive at Nabble.com.
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Slow Query
@ 2015-08-14 05:10 林士博 <lin@repica.co.jp>
parent: robbyc <robcampbell73@gmail.com>
0 siblings, 1 reply; 112+ messages in thread
From: 林士博 @ 2015-08-14 05:10 UTC (permalink / raw)
To: robbyc <robcampbell73@gmail.com>; +Cc: pgsql-performance
Is the "Vacancy"."ID" a primary key?
Or is unique in Vacancy table?
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Slow Query
@ 2015-08-14 06:09 林士博 <lin@repica.co.jp>
parent: 林士博 <lin@repica.co.jp>
0 siblings, 0 replies; 112+ messages in thread
From: 林士博 @ 2015-08-14 06:09 UTC (permalink / raw)
To: Robert Campbell <robcampbell73@gmail.com>; pgsql-performance
OK.
If you benchmark that correctly,
then it seems that adding some redundant search can get the query faster in
some special cases.
And without further info, I can not see any reason.
2015-08-14 14:35 GMT+09:00 Robert Campbell <robcampbell73@gmail.com>:
> Hi,
>
> My mistake, didnt apply the sub query properly the first time.
>
> It does return records but not quite as fast as original query, about
> 200ms slower
>
> Vacancy ID is a primary key.
>
> On Fri, Aug 14, 2015 at 3:10 PM, 林士博 <lin@repica.co.jp> wrote:
>
>> Is the "Vacancy"."ID" a primary key?
>> Or is unique in Vacancy table?
>>
>>
>
>
> --
> Regards
>
> Robert Campbell
> +61412062971
> robcampbell73@gmail.com
>
^ permalink raw reply [nested|flat] 112+ messages in thread
* slow query
@ 2020-04-03 16:03 dangal <danielito.gallo@gmail.com>
0 siblings, 1 reply; 112+ messages in thread
From: dangal @ 2020-04-03 16:03 UTC (permalink / raw)
To: pgsql-performance
Dear I have a question to ask you
I am having a slow problem with a query and I am seeing with the explain
that the current cost and time differ by 4 times
Postgres version 9.5.16 in centos 7.6
To try to solve this run the statistics to the table and the same problem
remains
It's a very big table 2 billion tuples
Do you have any idea what I can do to improve
Thank you very much for your time
Any data you need I can provide
I share a part of the explain
Hash Right Join (cost=11114339.65..12172907.42 rows=886647 width=158)
(actual time=1906344.617..1963668.889 rows=3362294 loops=1)"
" Output: ba.bus_ent_inst_id_auto, ba.att_value_1,
ba.att_value_num_1, att_call.ent_inst_att_str_value,
att_nro.ent_inst_att_str_value, att_bis.ent_inst_att_str_value,
att_pad.ent_inst_att_str_value, att_manz.ent_inst_att_str_value, att_a
(...)"
" Hash Cond: ((att_barr.env_id = ba.env_id) AND
(att_barr.bus_ent_inst_id = ba.bus_ent_inst_id_auto))"
" Buffers: shared hit=5814458 read=1033324 dirtied=790"
--
Sent from: https://www.postgresql-archive.org/PostgreSQL-performance-f2050081.html
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: slow query
@ 2020-04-03 16:07 Justin Pryzby <pryzby@telsasoft.com>
parent: dangal <danielito.gallo@gmail.com>
0 siblings, 1 reply; 112+ messages in thread
From: Justin Pryzby @ 2020-04-03 16:07 UTC (permalink / raw)
To: dangal <danielito.gallo@gmail.com>; +Cc: pgsql-performance
On Fri, Apr 03, 2020 at 09:03:49AM -0700, dangal wrote:
> Dear I have a question to ask you
> I am having a slow problem with a query and I am seeing with the explain that the current cost and time differ by 4 times
The "cost" is in arbitrary units, and the time is in units of milliseconds.
The cost is not an expected duration.
> Postgres version 9.5.16 in centos 7.6
> To try to solve this run the statistics to the table and the same problem
> remains
> It's a very big table 2 billion tuples
> Do you have any idea what I can do to improve
> Thank you very much for your time
> Any data you need I can provide
Please check here.
https://wiki.postgresql.org/wiki/Slow_Query_Questions
> I share a part of the explain
It's not very useful to see a fragment of it.
--
Justin
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: slow query
@ 2020-04-03 16:18 dangal <danielito.gallo@gmail.com>
parent: Justin Pryzby <pryzby@telsasoft.com>
0 siblings, 1 reply; 112+ messages in thread
From: dangal @ 2020-04-03 16:18 UTC (permalink / raw)
To: pgsql-performance
Justin thank you very much for your answer, as you can also see the number of
rows differs a lot
I attach the complete explain, do not attach it because it is large
"HashAggregate (cost=12640757.46..12713163.46 rows=385 width=720) (actual
time=1971962.023..1971962.155 rows=306 loops=1)"
" Output: bi.bus_ent_inst_name_num, bi.att_value_num_7, bi.att_value_10,
((SubPlan 1)), ((SubPlan 2)), a2.ent_inst_att_str_value, ba.att_value_1,
depto2.att_value_1, loc2.att_value_1, att_call.ent_inst_att_str_value,
att_nro.ent_inst_att_str_value, att_bis (...)"
" Group Key: bi.bus_ent_inst_name_num, bi.att_value_num_7, bi.att_value_10,
(SubPlan 1), (SubPlan 2), a2.ent_inst_att_str_value, ba.att_value_1,
depto2.att_value_1, loc2.att_value_1, att_call.ent_inst_att_str_value,
att_nro.ent_inst_att_str_value, att_bis. (...)"
" Buffers: shared hit=5817744 read=1034292 dirtied=790, local hit=2"
" -> Nested Loop (cost=11114347.52..12640740.13 rows=385 width=720)
(actual time=1906401.083..1971959.176 rows=306 loops=1)"
" Output: bi.bus_ent_inst_name_num, bi.att_value_num_7,
bi.att_value_10, (SubPlan 1), (SubPlan 2), a2.ent_inst_att_str_value,
ba.att_value_1, depto2.att_value_1, loc2.att_value_1,
att_call.ent_inst_att_str_value, att_nro.ent_inst_att_str_value, att_b
(...)"
" Buffers: shared hit=5817744 read=1034292 dirtied=790, local hit=2"
" -> Hash Join (cost=11114346.94..12228344.41 rows=1427 width=704)
(actual time=1906372.468..1964409.907 rows=306 loops=1)"
" Output: bi.bus_ent_inst_name_num, bi.att_value_num_7,
bi.att_value_10, ba.bus_ent_inst_id_auto, ba.att_value_1,
att_call.ent_inst_att_str_value, att_nro.ent_inst_att_str_value,
att_bis.ent_inst_att_str_value, att_pad.ent_inst_att_str_value, a (...)"
" Hash Cond: (ba.att_value_num_1 =
(bi.bus_ent_inst_name_num)::numeric)"
" Buffers: shared hit=5814458 read=1033324 dirtied=790, local
hit=2"
" -> Hash Right Join (cost=11114339.65..12172907.42
rows=886647 width=158) (actual time=1906344.617..1963668.889 rows=3362294
loops=1)"
" Output: ba.bus_ent_inst_id_auto, ba.att_value_1,
ba.att_value_num_1, att_call.ent_inst_att_str_value,
att_nro.ent_inst_att_str_value, att_bis.ent_inst_att_str_value,
att_pad.ent_inst_att_str_value, att_manz.ent_inst_att_str_value, att_a
(...)"
" Hash Cond: ((att_barr.env_id = ba.env_id) AND
(att_barr.bus_ent_inst_id = ba.bus_ent_inst_id_auto))"
" Buffers: shared hit=5814458 read=1033324 dirtied=790"
" -> Index Only Scan using ix_bus_ent_inst_attr_03 on
public.bus_ent_inst_attribute att_barr (cost=0.83..1024093.06 rows=4508264
width=24) (actual time=10.435..52888.091 rows=4244011 loops=1)"
" Output: att_barr.att_id,
att_barr.ent_inst_att_str_value, att_barr.env_id, att_barr.bus_ent_inst_id,
att_barr.reg_status"
" Index Cond: (att_barr.att_id = 1115)"
" Heap Fetches: 120577"
" Buffers: shared hit=503194 read=31197 dirtied=5"
" -> Hash (cost=11101039.12..11101039.12 rows=886647
width=146) (actual time=1906329.888..1906329.888 rows=3362294 loops=1)"
" Output: ba.bus_ent_inst_id_auto, ba.att_value_1,
ba.env_id, ba.att_value_num_1, att_call.ent_inst_att_str_value,
att_nro.ent_inst_att_str_value, att_bis.ent_inst_att_str_value,
att_pad.ent_inst_att_str_value, att_manz.ent_inst_att (...)"
" Buckets: 4194304 (originally 1048576) Batches: 1
(originally 1) Memory Usage: 396824kB"
" Buffers: shared hit=5311264 read=1002127
dirtied=785"
" -> Hash Right Join
(cost=10328938.09..11101039.12 rows=886647 width=146) (actual
time=1867557.718..1904218.946 rows=3362294 loops=1)"
" Output: ba.bus_ent_inst_id_auto,
ba.att_value_1, ba.env_id, ba.att_value_num_1,
att_call.ent_inst_att_str_value, att_nro.ent_inst_att_str_value,
att_bis.ent_inst_att_str_value, att_pad.ent_inst_att_str_value,
att_manz.ent_in (...)"
" Hash Cond: ((att_apt.env_id = ba.env_id)
AND (att_apt.bus_ent_inst_id = ba.bus_ent_inst_id_auto))"
" Buffers: shared hit=5311264 read=1002127
dirtied=785"
" -> Index Only Scan using
ix_bus_ent_inst_attr_03 on public.bus_ent_inst_attribute att_apt
(cost=0.83..746958.06 rows=3287982 width=24) (actual time=0.091..32788.731
rows=3491599 loops=1)"
" Output: att_apt.att_id,
att_apt.ent_inst_att_str_value, att_apt.env_id, att_apt.bus_ent_inst_id,
att_apt.reg_status"
" Index Cond: (att_apt.att_id = 1113)"
" Heap Fetches: 88910"
" Buffers: shared hit=178090 read=25341
dirtied=5"
" -> Hash (cost=10315637.55..10315637.55
rows=886647 width=130) (actual time=1867553.445..1867553.445 rows=3362294
loops=1)"
" Output: ba.bus_ent_inst_id_auto,
ba.att_value_1, ba.env_id, ba.att_value_num_1,
att_call.ent_inst_att_str_value, att_nro.ent_inst_att_str_value,
att_bis.ent_inst_att_str_value, att_pad.ent_inst_att_str_value, att_manz.
(...)"
" Buckets: 4194304 (originally 1048576)
Batches: 1 (originally 1) Memory Usage: 376885kB"
" Buffers: shared hit=5133174
read=976786 dirtied=780"
" -> Merge Left Join
(cost=10304076.40..10315637.55 rows=886647 width=130) (actual
time=1862979.687..1865773.765 rows=3362294 loops=1)"
" Output:
ba.bus_ent_inst_id_auto, ba.att_value_1, ba.env_id, ba.att_value_num_1,
att_call.ent_inst_att_str_value, att_nro.ent_inst_att_str_value,
att_bis.ent_inst_att_str_value, att_pad.ent_inst_att_str_value, att (...)"
" Merge Cond: ((ba.env_id =
loc2.env_id) AND (((att_loc_hecho.ent_inst_att_str_value)::integer) =
loc2.bus_ent_inst_name_num))"
" Buffers: shared hit=5133174
read=976786 dirtied=780"
" -> Sort
(cost=10178591.32..10180807.94 rows=886647 width=141) (actual
time=1862965.240..1863856.321 rows=3362294 loops=1)"
" Output:
ba.bus_ent_inst_id_auto, ba.att_value_1, ba.env_id, ba.att_value_num_1,
att_loc_hecho.ent_inst_att_str_value, att_call.ent_inst_att_str_value,
att_nro.ent_inst_att_str_value, att_bis.ent_inst_att_st (...)"
" Sort Key: ba.env_id,
((att_loc_hecho.ent_inst_att_str_value)::integer)"
" Sort Method: quicksort
Memory: 544870kB"
" Buffers: shared
hit=5133062 read=976781 dirtied=780"
" -> Merge Left Join
(cost=10079438.31..10090999.47 rows=886647 width=141) (actual
time=1854085.484..1857592.771 rows=3362294 loops=1)"
" Output:
ba.bus_ent_inst_id_auto, ba.att_value_1, ba.env_id, ba.att_value_num_1,
att_loc_hecho.ent_inst_att_str_value, att_call.ent_inst_att_str_value,
att_nro.ent_inst_att_str_value, att_bis.ent_inst_ (...)"
" Merge Cond:
((ba.env_id = depto2.env_id) AND
(((att_dir_hecho.ent_inst_att_str_value)::integer) =
depto2.bus_ent_inst_name_num))"
" Buffers: shared
hit=5133062 read=976781 dirtied=780"
" -> Sort
(cost=9953953.24..9956169.85 rows=886647 width=152) (actual
time=1854079.630..1855329.406 rows=3362294 loops=1)"
" Output:
ba.bus_ent_inst_id_auto, ba.att_value_1, ba.env_id, ba.att_value_num_1,
att_dir_hecho.ent_inst_att_str_value, att_loc_hecho.ent_inst_att_str_value,
att_call.ent_inst_att_str_value, att_n (...)"
" Sort Key:
ba.env_id, ((att_dir_hecho.ent_inst_att_str_value)::integer)"
" Sort Method:
quicksort Memory: 544857kB"
" Buffers:
shared hit=5133055 read=976779 dirtied=780"
" -> Hash
Right Join (cost=9791232.05..9866361.38 rows=886647 width=152) (actual
time=1844734.652..1849217.758 rows=3362294 loops=1)"
" Output:
ba.bus_ent_inst_id_auto, ba.att_value_1, ba.env_id, ba.att_value_num_1,
att_dir_hecho.ent_inst_att_str_value, att_loc_hecho.ent_inst_att_str_value,
att_call.ent_inst_att_str_value, (...)"
" Hash
Cond: ((att_rut.env_id = ba.env_id) AND (att_rut.bus_ent_inst_id =
ba.bus_ent_inst_id_auto))"
"
Buffers: shared hit=5133055 read=976779 dirtied=780"
" ->
Index Only Scan using ix_bus_ent_inst_attr_03 on
public.bus_ent_inst_attribute att_rut (cost=0.83..72690.43 rows=319036
width=24) (actual time=17.325..3078.312 rows=149644 loops=1)"
"
Output: att_rut.att_id, att_rut.ent_inst_att_str_value, att_rut.env_id,
att_rut.bus_ent_inst_id, att_rut.reg_status"
"
Index Cond: (att_rut.att_id = 1138)"
"
Heap Fetches: 5299"
"
Buffers: shared hit=26350 read=1137"
" ->
Hash (cost=9777931.51..9777931.51 rows=886647 width=136) (actual
time=1844713.350..1844713.350 rows=3362294 loops=1)"
"
Output: ba.bus_ent_inst_id_auto, ba.att_value_1, ba.env_id,
ba.att_value_num_1, att_dir_hecho.ent_inst_att_str_value,
att_loc_hecho.ent_inst_att_str_value, att_call.ent_inst_att_str_ (...)"
"
Buckets: 4194304 (originally 1048576) Batches: 1 (originally 1) Memory
Usage: 329015kB"
"
Buffers: shared hit=5106705 read=975642 dirtied=780"
"
-> Hash Right Join (cost=9705206.15..9777931.51 rows=886647 width=136)
(actual time=1837569.880..1842945.853 rows=3362294 loops=1)"
"
Output: ba.bus_ent_inst_id_auto, ba.att_value_1, ba.env_id,
ba.att_value_num_1, att_dir_hecho.ent_inst_att_str_value,
att_loc_hecho.ent_inst_att_str_value, att_call.ent_inst_at (...)"
"
Hash Cond: ((att_km.env_id = ba.env_id) AND (att_km.bus_ent_inst_id =
ba.bus_ent_inst_id_auto))"
"
Buffers: shared hit=5106705 read=975642 dirtied=780"
"
-> Index Only Scan using ix_bus_ent_inst_attr_02 on
public.bus_ent_inst_attribute att_km (cost=0.70..70286.34 rows=319036
width=13) (actual time=0.107..2995.494 rows=149942 l (...)"
"
Output: att_km.att_id, att_km.ent_inst_att_num_value, att_km.env_id,
att_km.bus_ent_inst_id, att_km.reg_status"
"
Index Cond: (att_km.att_id = 1132)"
"
Heap Fetches: 5330"
"
Buffers: shared hit=59470 read=1171"
"
-> Hash (cost=9691905.74..9691905.74 rows=886647 width=131) (actual
time=1837565.949..1837565.949 rows=3362294 loops=1)"
"
Output: ba.bus_ent_inst_id_auto, ba.att_value_1, ba.env_id,
ba.att_value_num_1, att_dir_hecho.ent_inst_att_str_value,
att_loc_hecho.ent_inst_att_str_value, att_call.ent_i (...)"
"
Buckets: 4194304 (originally 1048576) Batches: 1 (originally 1) Memory
Usage: 328650kB"
"
Buffers: shared hit=5047235 read=974471 dirtied=780"
"
-> Hash Right Join (cost=7694366.79..9691905.74 rows=886647 width=131)
(actual time=1710903.369..1834807.221 rows=3362294 loops=1)"
"
Output: ba.bus_ent_inst_id_auto, ba.att_value_1, ba.env_id,
ba.att_value_num_1, att_dir_hecho.ent_inst_att_str_value,
att_loc_hecho.ent_inst_att_str_value, att_call (...)"
"
Hash Cond: ((att_bis.env_id = ba.env_id) AND (att_bis.bus_ent_inst_id =
ba.bus_ent_inst_id_auto))"
"
Buffers: shared hit=5047235 read=974471 dirtied=780"
"
-> Index Only Scan using ix_bus_ent_inst_attr_03 on
public.bus_ent_inst_attribute att_bis (cost=0.83..1932476.93 rows=8508077
width=24) (actual time=6.488..116892 (...)"
"
Output: att_bis.att_id, att_bis.ent_inst_att_str_value, att_bis.env_id,
att_bis.bus_ent_inst_id, att_bis.reg_status"
"
Index Cond: (att_bis.att_id = 1117)"
"
Heap Fetches: 228123"
"
Buffers: shared hit=218185 read=52064 dirtied=27"
"
-> Hash (cost=7681066.26..7681066.26 rows=886647 width=115) (actual
time=1710893.007..1710893.007 rows=3362294 loops=1)"
"
Output: ba.bus_ent_inst_id_auto, ba.att_value_1, ba.env_id,
ba.att_value_num_1, att_dir_hecho.ent_inst_att_str_value,
att_loc_hecho.ent_inst_att_str_value, at (...)"
"
Buckets: 4194304 (originally 1048576) Batches: 1 (originally 1) Memory
Usage: 309513kB"
"
Buffers: shared hit=4829050 read=922407 dirtied=753"
"
-> Hash Right Join (cost=5969990.07..7681066.26 rows=886647 width=115)
(actual time=1566042.427..1708291.649 rows=3362294 loops=1)"
"
Output: ba.bus_ent_inst_id_auto, ba.att_value_1, ba.env_id,
ba.att_value_num_1, att_dir_hecho.ent_inst_att_str_value,
att_loc_hecho.ent_inst_att_str_val (...)"
"
Hash Cond: ((att_call.env_id = ba.env_id) AND (att_call.bus_ent_inst_id =
ba.bus_ent_inst_id_auto))"
"
Buffers: shared hit=4829050 read=922407 dirtied=753"
"
-> Index Only Scan using ix_bus_ent_inst_attr_03 on
public.bus_ent_inst_attribute att_call (cost=0.83..1655345.90 rows=7287794
width=24) (actual time= (...)"
"
Output: att_call.att_id, att_call.ent_inst_att_str_value, att_call.env_id,
att_call.bus_ent_inst_id, att_call.reg_status"
"
Index Cond: (att_call.att_id = 1119)"
"
Heap Fetches: 213801"
"
Buffers: shared hit=1852588 read=60151 dirtied=23"
"
-> Hash (cost=5956689.54..5956689.54 rows=886647 width=99) (actual
time=1566015.832..1566015.832 rows=3362294 loops=1)"
"
Output: ba.bus_ent_inst_id_auto, ba.att_value_1, ba.env_id,
ba.att_value_num_1, att_dir_hecho.ent_inst_att_str_value,
att_loc_hecho.ent_inst_att_s (...)"
"
Buckets: 4194304 (originally 1048576) Batches: 1 (originally 1) Memory
Usage: 258291kB"
"
Buffers: shared hit=2976462 read=862256 dirtied=730"
"
-> Hash Right Join (cost=4253571.63..5956689.54 rows=886647 width=99)
(actual time=1355922.435..1563760.249 rows=3362294 loops=1)"
"
Output: ba.bus_ent_inst_id_auto, ba.att_value_1, ba.env_id,
ba.att_value_num_1, att_dir_hecho.ent_inst_att_str_value,
att_loc_hecho.ent_inst (...)"
"
Hash Cond: ((att_dir_hecho.env_id = ba.env_id) AND
(att_dir_hecho.bus_ent_inst_id = ba.bus_ent_inst_id_auto))"
"
Buffers: shared hit=2976462 read=862256 dirtied=730"
"
-> Index Only Scan using ix_bus_ent_inst_attr_03 on
public.bus_ent_inst_attribute att_dir_hecho (cost=0.83..1647646.84
rows=7253898 width= (...)"
"
Output: att_dir_hecho.att_id, att_dir_hecho.ent_inst_att_str_value,
att_dir_hecho.env_id, att_dir_hecho.bus_ent_inst_id, att_dir_hecho (...)"
"
Index Cond: (att_dir_hecho.att_id = 1122)"
"
Heap Fetches: 221189"
"
Buffers: shared hit=217265 read=76872 dirtied=96"
"
-> Hash (cost=4240271.10..4240271.10 rows=886647 width=83) (actual
time=1355910.157..1355910.157 rows=3362294 loops=1)"
"
Output: ba.bus_ent_inst_id_auto, ba.att_value_1, ba.env_id,
ba.att_value_num_1, att_loc_hecho.ent_inst_att_str_value, att_nro.ent_inst
(...)"
"
Buckets: 4194304 (originally 1048576) Batches: 1 (originally 1) Memory
Usage: 224784kB"
"
Buffers: shared hit=2759197 read=785384 dirtied=634"
"
-> Hash Right Join (cost=2672428.25..4240271.10 rows=886647 width=83)
(actual time=1097647.410..1353630.001 rows=3362294 loops=1)"
"
Output: ba.bus_ent_inst_id_auto, ba.att_value_1, ba.env_id,
ba.att_value_num_1, att_loc_hecho.ent_inst_att_str_value, att_nro.en (...)"
"
Hash Cond: ((att_loc_hecho.env_id = ba.env_id) AND
(att_loc_hecho.bus_ent_inst_id = ba.bus_ent_inst_id_auto))"
"
Buffers: shared hit=2759197 read=785384 dirtied=634"
"
-> Index Only Scan using ix_bus_ent_inst_attr_03 on
public.bus_ent_inst_attribute att_loc_hecho (cost=0.83..1516778.41 rows=66
(...)"
"
Output: att_loc_hecho.att_id, att_loc_hecho.ent_inst_att_str_value,
att_loc_hecho.env_id, att_loc_hecho.bus_ent_inst_id, a (...)"
"
Index Cond: (att_loc_hecho.att_id = 1133)"
"
Heap Fetches: 218787"
"
Buffers: shared hit=332968 read=93935 dirtied=115"
"
-> Hash (cost=2659127.72..2659127.72 rows=886647 width=67) (actual
time=1097642.027..1097642.027 rows=3362294 loops=1)"
"
Output: ba.bus_ent_inst_id_auto, ba.att_value_1, ba.env_id,
ba.att_value_num_1, att_nro.ent_inst_att_str_value, att_pad.en (...)"
"
Buckets: 4194304 (originally 1048576) Batches: 1 (originally 1) Memory
Usage: 215839kB"
"
Buffers: shared hit=2426229 read=691449 dirtied=519"
"
-> Hash Right Join (cost=1353880.71..2659127.72 rows=886647 width=67)
(actual time=466534.722..1095259.942 rows=3362294 (...)"
"
Output: ba.bus_ent_inst_id_auto, ba.att_value_1, ba.env_id,
ba.att_value_num_1, att_nro.ent_inst_att_str_value, att_ (...)"
"
Hash Cond: ((att_nro.env_id = ba.env_id) AND (att_nro.bus_ent_inst_id =
ba.bus_ent_inst_id_auto))"
"
Buffers: shared hit=2426229 read=691449 dirtied=519"
"
-> Index Only Scan using ix_bus_ent_inst_attr_03 on
public.bus_ent_inst_attribute att_nro (cost=0.83..1262736.66 r (...)"
"
Output: att_nro.att_id, att_nro.ent_inst_att_str_value, att_nro.env_id,
att_nro.bus_ent_inst_id, att_nro.reg_s (...)"
"
Index Cond: (att_nro.att_id = 1135)"
"
Heap Fetches: 156988"
"
Buffers: shared hit=1568458 read=151792 dirtied=285"
"
-> Hash (cost=1340580.18..1340580.18 rows=886647 width=51) (actual
time=466528.985..466528.985 rows=3362294 loops= (...)"
"
Output: ba.bus_ent_inst_id_auto, ba.att_value_1, ba.env_id,
ba.att_value_num_1, att_pad.ent_inst_att_str_value (...)"
"
Buckets: 4194304 (originally 1048576) Batches: 1 (originally 1) Memory
Usage: 204115kB"
"
Buffers: shared hit=857771 read=539657 dirtied=234"
"
-> Hash Right Join (cost=1265450.85..1340580.18 rows=886647 width=51)
(actual time=464578.744..465343.707 ro (...)"
"
Output: ba.bus_ent_inst_id_auto, ba.att_value_1, ba.env_id,
ba.att_value_num_1, att_pad.ent_inst_att_str (...)"
"
Hash Cond: ((att_manz.env_id = ba.env_id) AND (att_manz.bus_ent_inst_id =
ba.bus_ent_inst_id_auto))"
"
Buffers: shared hit=857771 read=539657 dirtied=234"
"
-> Index Only Scan using ix_bus_ent_inst_attr_03 on
public.bus_ent_inst_attribute att_manz (cost=0.83. (...)"
"
Output: att_manz.att_id, att_manz.ent_inst_att_str_value, att_manz.env_id,
att_manz.bus_ent_inst_i (...)"
"
Index Cond: (att_manz.att_id = 1134)"
"
Heap Fetches: 14"
"
Buffers: shared hit=276 read=15"
"
-> Hash (cost=1252150.32..1252150.32 rows=886647 width=35) (actual
time=464569.271..464569.271 rows=33 (...)"
"
Output: ba.bus_ent_inst_id_auto, ba.att_value_1, ba.env_id,
ba.att_value_num_1, att_pad.ent_inst_a (...)"
"
Buckets: 4194304 (originally 1048576) Batches: 1 (originally 1) Memory
Usage: 204114kB"
"
Buffers: shared hit=857495 read=539642 dirtied=234"
"
-> Hash Right Join (cost=1177020.99..1252150.32 rows=886647 width=35)
(actual time=184587.973..4 (...)"
"
Output: ba.bus_ent_inst_id_auto, ba.att_value_1, ba.env_id,
ba.att_value_num_1, att_pad.ent_ (...)"
"
Hash Cond: ((att_pad.env_id = ba.env_id) AND (att_pad.bus_ent_inst_id =
ba.bus_ent_inst_id_a (...)"
"
Buffers: shared hit=857495 read=539642 dirtied=234"
"
-> Index Only Scan using ix_bus_ent_inst_attr_03 on
public.bus_ent_inst_attribute att_pad (...)"
"
Output: att_pad.att_id, att_pad.ent_inst_att_str_value, att_pad.env_id,
att_pad.bus_en (...)"
"
Index Cond: (att_pad.att_id = 1136)"
"
Heap Fetches: 54024"
"
Buffers: shared hit=334762 read=60835 dirtied=136"
"
-> Hash (cost=1163720.45..1163720.45 rows=886647 width=19) (actual
time=184573.023..184573 (...)"
"
Output: ba.bus_ent_inst_id_auto, ba.att_value_1, ba.env_id,
ba.att_value_num_1"
"
Buckets: 4194304 (originally 1048576) Batches: 1 (originally 1) Memory
Usage: 200216 (...)"
"
Buffers: shared hit=522733 read=478807 dirtied=98"
"
-> Bitmap Heap Scan on public.bus_ent_instance ba
(cost=35242.83..1163720.45 rows=88 (...)"
"
Output: ba.bus_ent_inst_id_auto, ba.att_value_1, ba.env_id,
ba.att_value_num_1"
"
Recheck Cond: ((((ba.bus_ent_inst_name_pre)::text = 'FOTPER'::text) AND
((ba.att (...)"
"
Heap Blocks: exact=981056"
"
Buffers: shared hit=522733 read=478807 dirtied=98"
"
-> BitmapOr (cost=35242.83..35242.83 rows=896239 width=0) (actual
time=33401.6 (...)"
"
Buffers: shared hit=43 read=20441"
"
-> Bitmap Index Scan on ix_bus_ent_instance_atts_namenum_01 (cost=0.00..
(...)"
"
Index Cond: (((ba.bus_ent_inst_name_pre)::text = 'FOTPER'::text) AND (...)"
"
Buffers: shared hit=9 read=5030"
"
-> Bitmap Index Scan on ix_bus_ent_instance_atts_namenum_01 (cost=0.00..
(...)"
"
Index Cond: (((ba.bus_ent_inst_name_pre)::text = 'FOTPER'::text) AND (...)"
"
Buffers: shared hit=34 read=15411"
" -> Sort
(cost=125485.07..125573.87 rows=35520 width=13) (actual time=5.831..312.164
rows=3217523 loops=1)"
" Output:
depto2.att_value_1, depto2.env_id, depto2.bus_ent_inst_name_num"
" Sort Key:
depto2.env_id, depto2.bus_ent_inst_name_num"
" Sort Method:
quicksort Memory: 26kB"
" Buffers:
shared hit=7 read=2"
" -> Bitmap
Heap Scan on public.bus_ent_instance depto2 (cost=971.85..122800.41
rows=35520 width=13) (actual time=5.758..5.776 rows=21 loops=1)"
" Output:
depto2.att_value_1, depto2.env_id, depto2.bus_ent_inst_name_num"
" Recheck
Cond: (depto2.bus_ent_id = 1091)"
" Heap
Blocks: exact=5"
"
Buffers: shared hit=7 read=2"
" ->
Bitmap Index Scan on ix_bus_ent_instance_01 (cost=0.00..962.97 rows=35520
width=0) (actual time=5.733..5.733 rows=21 loops=1)"
"
Index Cond: (depto2.bus_ent_id = 1091)"
"
Buffers: shared hit=2 read=2"
" -> Sort
(cost=125485.07..125573.87 rows=35520 width=13) (actual time=14.418..320.637
rows=3217335 loops=1)"
" Output: loc2.att_value_1,
loc2.env_id, loc2.bus_ent_inst_name_num"
" Sort Key: loc2.env_id,
loc2.bus_ent_inst_name_num"
" Sort Method: quicksort
Memory: 76kB"
" Buffers: shared hit=112
read=5"
" -> Bitmap Heap Scan on
public.bus_ent_instance loc2 (cost=971.85..122800.41 rows=35520 width=13)
(actual time=13.305..13.922 rows=725 loops=1)"
" Output:
loc2.att_value_1, loc2.env_id, loc2.bus_ent_inst_name_num"
" Recheck Cond:
(loc2.bus_ent_id = 1165)"
" Heap Blocks:
exact=110"
" Buffers: shared
hit=112 read=5"
" -> Bitmap Index
Scan on ix_bus_ent_instance_01 (cost=0.00..962.97 rows=35520 width=0)
(actual time=13.262..13.262 rows=725 loops=1)"
" Index Cond:
(loc2.bus_ent_id = 1165)"
" Buffers:
shared hit=2 read=5"
" -> Hash (cost=4.35..4.35 rows=235 width=552) (actual
time=0.175..0.175 rows=235 loops=1)"
" Output: bi.bus_ent_inst_name_num, bi.att_value_num_7,
bi.att_value_10"
" Buckets: 1024 Batches: 1 Memory Usage: 19kB"
" Buffers: local hit=2"
" -> Seq Scan on pg_temp_179.temp_table bi
(cost=0.00..4.35 rows=235 width=552) (actual time=0.015..0.055 rows=235
loops=1)"
" Output: bi.bus_ent_inst_name_num,
bi.att_value_num_7, bi.att_value_10"
" Buffers: local hit=2"
" -> Index Scan using ix_bus_ent_inst_attr_01 on
public.bus_ent_inst_attribute a2 (cost=0.58..237.03 rows=123 width=20)
(actual time=23.167..23.168 rows=1 loops=306)"
" Output: a2.env_id, a2.bus_ent_inst_id, a2.att_id,
a2.att_row_id_auto, a2.att_index_id, a2.ent_inst_att_num_value,
a2.ent_inst_att_str_value, a2.ent_inst_att_dte_value,
a2.ent_inst_att_doc_id, a2.ent_inst_att_tran_1, a2.ent_inst_att_tran_2, a2
(...)"
" Index Cond: ((a2.bus_ent_inst_id = ba.bus_ent_inst_id_auto)
AND (a2.att_id = 1083))"
" Buffers: shared hit=635 read=895"
" SubPlan 1"
" -> Index Scan using ix_bus_ent_inst_attr_01 on
public.bus_ent_inst_attribute a (cost=0.58..141.91 rows=72 width=16)
(actual time=0.646..0.647 rows=1 loops=306)"
" Output: a.ent_inst_att_str_value"
" Index Cond: ((ba.bus_ent_inst_id_auto = a.bus_ent_inst_id)
AND (a.att_id = 1071))"
" Filter: (a.reg_status = 0)"
" Buffers: shared hit=1434 read=31"
" SubPlan 2"
" -> Index Scan using ix_bus_ent_inst_attr_01 on
public.bus_ent_inst_attribute t (cost=0.58..46.15 rows=21 width=16) (actual
time=0.839..0.841 rows=0 loops=306)"
" Output: t.ent_inst_att_str_value"
" Index Cond: ((ba.bus_ent_inst_id_auto = t.bus_ent_inst_id)
AND (t.att_id = 1141))"
" Filter: (t.reg_status = 0)"
" Buffers: shared hit=1217 read=42"
"Planning time: 18.329 ms"
"Execution time: 1972336.524 ms"
--
Sent from: https://www.postgresql-archive.org/PostgreSQL-performance-f2050081.html
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: slow query
@ 2020-04-07 10:20 Michael Christofides <michristofides@gmail.com>
parent: dangal <danielito.gallo@gmail.com>
0 siblings, 0 replies; 112+ messages in thread
From: Michael Christofides @ 2020-04-07 10:20 UTC (permalink / raw)
To: dangal <danielito.gallo@gmail.com>; +Cc: pgsql-performance
That plan looks like it might have been cropped in places, and the
formatting is making it tricky to help.
Could you try again, pasting the plan into https://explain.depesz.com/ to
make it easier to review?
On Fri, Apr 3, 2020 at 5:18 PM dangal <danielito.gallo@gmail.com> wrote:
> Justin thank you very much for your answer, as you can also see the number
> of
> rows differs a lot
> I attach the complete explain, do not attach it because it is large
>
> "HashAggregate (cost=12640757.46..12713163.46 rows=385 width=720) (actual
> time=1971962.023..1971962.155 rows=306 loops=1)"
> " Output: bi.bus_ent_inst_name_num, bi.att_value_num_7, bi.att_value_10,
> ((SubPlan 1)), ((SubPlan 2)), a2.ent_inst_att_str_value, ba.att_value_1,
> depto2.att_value_1, loc2.att_value_1, att_call.ent_inst_att_str_value,
> att_nro.ent_inst_att_str_value, att_bis (...)"
> " Group Key: bi.bus_ent_inst_name_num, bi.att_value_num_7,
> bi.att_value_10,
> (SubPlan 1), (SubPlan 2), a2.ent_inst_att_str_value, ba.att_value_1,
> depto2.att_value_1, loc2.att_value_1, att_call.ent_inst_att_str_value,
> att_nro.ent_inst_att_str_value, att_bis. (...)"
> " Buffers: shared hit=5817744 read=1034292 dirtied=790, local hit=2"
> " -> Nested Loop (cost=11114347.52..12640740.13 rows=385 width=720)
> (actual time=1906401.083..1971959.176 rows=306 loops=1)"
> " Output: bi.bus_ent_inst_name_num, bi.att_value_num_7,
> bi.att_value_10, (SubPlan 1), (SubPlan 2), a2.ent_inst_att_str_value,
> ba.att_value_1, depto2.att_value_1, loc2.att_value_1,
> att_call.ent_inst_att_str_value, att_nro.ent_inst_att_str_value, att_b
> (...)"
> " Buffers: shared hit=5817744 read=1034292 dirtied=790, local hit=2"
> " -> Hash Join (cost=11114346.94..12228344.41 rows=1427 width=704)
> (actual time=1906372.468..1964409.907 rows=306 loops=1)"
> " Output: bi.bus_ent_inst_name_num, bi.att_value_num_7,
> bi.att_value_10, ba.bus_ent_inst_id_auto, ba.att_value_1,
> att_call.ent_inst_att_str_value, att_nro.ent_inst_att_str_value,
> att_bis.ent_inst_att_str_value, att_pad.ent_inst_att_str_value, a (...)"
> " Hash Cond: (ba.att_value_num_1 =
> (bi.bus_ent_inst_name_num)::numeric)"
> " Buffers: shared hit=5814458 read=1033324 dirtied=790, local
> hit=2"
> " -> Hash Right Join (cost=11114339.65..12172907.42
> rows=886647 width=158) (actual time=1906344.617..1963668.889 rows=3362294
> loops=1)"
> " Output: ba.bus_ent_inst_id_auto, ba.att_value_1,
> ba.att_value_num_1, att_call.ent_inst_att_str_value,
> att_nro.ent_inst_att_str_value, att_bis.ent_inst_att_str_value,
> att_pad.ent_inst_att_str_value, att_manz.ent_inst_att_str_value, att_a
> (...)"
> " Hash Cond: ((att_barr.env_id = ba.env_id) AND
> (att_barr.bus_ent_inst_id = ba.bus_ent_inst_id_auto))"
> " Buffers: shared hit=5814458 read=1033324 dirtied=790"
> " -> Index Only Scan using ix_bus_ent_inst_attr_03 on
> public.bus_ent_inst_attribute att_barr (cost=0.83..1024093.06 rows=4508264
> width=24) (actual time=10.435..52888.091 rows=4244011 loops=1)"
> " Output: att_barr.att_id,
> att_barr.ent_inst_att_str_value, att_barr.env_id, att_barr.bus_ent_inst_id,
> att_barr.reg_status"
> " Index Cond: (att_barr.att_id = 1115)"
> " Heap Fetches: 120577"
> " Buffers: shared hit=503194 read=31197 dirtied=5"
> " -> Hash (cost=11101039.12..11101039.12 rows=886647
> width=146) (actual time=1906329.888..1906329.888 rows=3362294 loops=1)"
> " Output: ba.bus_ent_inst_id_auto, ba.att_value_1,
> ba.env_id, ba.att_value_num_1, att_call.ent_inst_att_str_value,
> att_nro.ent_inst_att_str_value, att_bis.ent_inst_att_str_value,
> att_pad.ent_inst_att_str_value, att_manz.ent_inst_att (...)"
> " Buckets: 4194304 (originally 1048576) Batches:
> 1
> (originally 1) Memory Usage: 396824kB"
> " Buffers: shared hit=5311264 read=1002127
> dirtied=785"
> " -> Hash Right Join
> (cost=10328938.09..11101039.12 rows=886647 width=146) (actual
> time=1867557.718..1904218.946 rows=3362294 loops=1)"
> " Output: ba.bus_ent_inst_id_auto,
> ba.att_value_1, ba.env_id, ba.att_value_num_1,
> att_call.ent_inst_att_str_value, att_nro.ent_inst_att_str_value,
> att_bis.ent_inst_att_str_value, att_pad.ent_inst_att_str_value,
> att_manz.ent_in (...)"
> " Hash Cond: ((att_apt.env_id = ba.env_id)
> AND (att_apt.bus_ent_inst_id = ba.bus_ent_inst_id_auto))"
> " Buffers: shared hit=5311264 read=1002127
> dirtied=785"
> " -> Index Only Scan using
> ix_bus_ent_inst_attr_03 on public.bus_ent_inst_attribute att_apt
> (cost=0.83..746958.06 rows=3287982 width=24) (actual time=0.091..32788.731
> rows=3491599 loops=1)"
> " Output: att_apt.att_id,
> att_apt.ent_inst_att_str_value, att_apt.env_id, att_apt.bus_ent_inst_id,
> att_apt.reg_status"
> " Index Cond: (att_apt.att_id = 1113)"
> " Heap Fetches: 88910"
> " Buffers: shared hit=178090
> read=25341
> dirtied=5"
> " -> Hash (cost=10315637.55..10315637.55
> rows=886647 width=130) (actual time=1867553.445..1867553.445 rows=3362294
> loops=1)"
> " Output: ba.bus_ent_inst_id_auto,
> ba.att_value_1, ba.env_id, ba.att_value_num_1,
> att_call.ent_inst_att_str_value, att_nro.ent_inst_att_str_value,
> att_bis.ent_inst_att_str_value, att_pad.ent_inst_att_str_value, att_manz.
> (...)"
> " Buckets: 4194304 (originally
> 1048576)
> Batches: 1 (originally 1) Memory Usage: 376885kB"
> " Buffers: shared hit=5133174
> read=976786 dirtied=780"
> " -> Merge Left Join
> (cost=10304076.40..10315637.55 rows=886647 width=130) (actual
> time=1862979.687..1865773.765 rows=3362294 loops=1)"
> " Output:
> ba.bus_ent_inst_id_auto, ba.att_value_1, ba.env_id, ba.att_value_num_1,
> att_call.ent_inst_att_str_value, att_nro.ent_inst_att_str_value,
> att_bis.ent_inst_att_str_value, att_pad.ent_inst_att_str_value, att (...)"
> " Merge Cond: ((ba.env_id =
> loc2.env_id) AND (((att_loc_hecho.ent_inst_att_str_value)::integer) =
> loc2.bus_ent_inst_name_num))"
> " Buffers: shared hit=5133174
> read=976786 dirtied=780"
> " -> Sort
> (cost=10178591.32..10180807.94 rows=886647 width=141) (actual
> time=1862965.240..1863856.321 rows=3362294 loops=1)"
> " Output:
> ba.bus_ent_inst_id_auto, ba.att_value_1, ba.env_id, ba.att_value_num_1,
> att_loc_hecho.ent_inst_att_str_value, att_call.ent_inst_att_str_value,
> att_nro.ent_inst_att_str_value, att_bis.ent_inst_att_st (...)"
> " Sort Key: ba.env_id,
> ((att_loc_hecho.ent_inst_att_str_value)::integer)"
> " Sort Method: quicksort
> Memory: 544870kB"
> " Buffers: shared
> hit=5133062 read=976781 dirtied=780"
> " -> Merge Left Join
> (cost=10079438.31..10090999.47 rows=886647 width=141) (actual
> time=1854085.484..1857592.771 rows=3362294 loops=1)"
> " Output:
> ba.bus_ent_inst_id_auto, ba.att_value_1, ba.env_id, ba.att_value_num_1,
> att_loc_hecho.ent_inst_att_str_value, att_call.ent_inst_att_str_value,
> att_nro.ent_inst_att_str_value, att_bis.ent_inst_ (...)"
> " Merge Cond:
> ((ba.env_id = depto2.env_id) AND
> (((att_dir_hecho.ent_inst_att_str_value)::integer) =
> depto2.bus_ent_inst_name_num))"
> " Buffers: shared
> hit=5133062 read=976781 dirtied=780"
> " -> Sort
> (cost=9953953.24..9956169.85 rows=886647 width=152) (actual
> time=1854079.630..1855329.406 rows=3362294 loops=1)"
> " Output:
> ba.bus_ent_inst_id_auto, ba.att_value_1, ba.env_id, ba.att_value_num_1,
> att_dir_hecho.ent_inst_att_str_value, att_loc_hecho.ent_inst_att_str_value,
> att_call.ent_inst_att_str_value, att_n (...)"
> " Sort Key:
> ba.env_id, ((att_dir_hecho.ent_inst_att_str_value)::integer)"
> " Sort Method:
> quicksort Memory: 544857kB"
> " Buffers:
> shared hit=5133055 read=976779 dirtied=780"
> " -> Hash
> Right Join (cost=9791232.05..9866361.38 rows=886647 width=152) (actual
> time=1844734.652..1849217.758 rows=3362294 loops=1)"
> "
> Output:
> ba.bus_ent_inst_id_auto, ba.att_value_1, ba.env_id, ba.att_value_num_1,
> att_dir_hecho.ent_inst_att_str_value, att_loc_hecho.ent_inst_att_str_value,
> att_call.ent_inst_att_str_value, (...)"
> " Hash
> Cond: ((att_rut.env_id = ba.env_id) AND (att_rut.bus_ent_inst_id =
> ba.bus_ent_inst_id_auto))"
> "
> Buffers: shared hit=5133055 read=976779 dirtied=780"
> " ->
> Index Only Scan using ix_bus_ent_inst_attr_03 on
> public.bus_ent_inst_attribute att_rut (cost=0.83..72690.43 rows=319036
> width=24) (actual time=17.325..3078.312 rows=149644 loops=1)"
> "
> Output: att_rut.att_id, att_rut.ent_inst_att_str_value, att_rut.env_id,
> att_rut.bus_ent_inst_id, att_rut.reg_status"
> "
> Index Cond: (att_rut.att_id = 1138)"
> "
> Heap Fetches: 5299"
> "
> Buffers: shared hit=26350 read=1137"
> " ->
> Hash (cost=9777931.51..9777931.51 rows=886647 width=136) (actual
> time=1844713.350..1844713.350 rows=3362294 loops=1)"
> "
> Output: ba.bus_ent_inst_id_auto, ba.att_value_1, ba.env_id,
> ba.att_value_num_1, att_dir_hecho.ent_inst_att_str_value,
> att_loc_hecho.ent_inst_att_str_value, att_call.ent_inst_att_str_ (...)"
> "
> Buckets: 4194304 (originally 1048576) Batches: 1 (originally 1) Memory
> Usage: 329015kB"
> "
> Buffers: shared hit=5106705 read=975642 dirtied=780"
> "
> -> Hash Right Join (cost=9705206.15..9777931.51 rows=886647 width=136)
> (actual time=1837569.880..1842945.853 rows=3362294 loops=1)"
> "
>
> Output: ba.bus_ent_inst_id_auto, ba.att_value_1, ba.env_id,
> ba.att_value_num_1, att_dir_hecho.ent_inst_att_str_value,
> att_loc_hecho.ent_inst_att_str_value, att_call.ent_inst_at (...)"
> "
>
> Hash Cond: ((att_km.env_id = ba.env_id) AND (att_km.bus_ent_inst_id =
> ba.bus_ent_inst_id_auto))"
> "
>
> Buffers: shared hit=5106705 read=975642 dirtied=780"
> "
>
> -> Index Only Scan using ix_bus_ent_inst_attr_02 on
> public.bus_ent_inst_attribute att_km (cost=0.70..70286.34 rows=319036
> width=13) (actual time=0.107..2995.494 rows=149942 l (...)"
> "
>
> Output: att_km.att_id, att_km.ent_inst_att_num_value, att_km.env_id,
> att_km.bus_ent_inst_id, att_km.reg_status"
> "
>
> Index Cond: (att_km.att_id = 1132)"
> "
>
> Heap Fetches: 5330"
> "
>
> Buffers: shared hit=59470 read=1171"
> "
>
> -> Hash (cost=9691905.74..9691905.74 rows=886647 width=131) (actual
> time=1837565.949..1837565.949 rows=3362294 loops=1)"
> "
>
> Output: ba.bus_ent_inst_id_auto, ba.att_value_1, ba.env_id,
> ba.att_value_num_1, att_dir_hecho.ent_inst_att_str_value,
> att_loc_hecho.ent_inst_att_str_value, att_call.ent_i (...)"
> "
>
> Buckets: 4194304 (originally 1048576) Batches: 1 (originally 1) Memory
> Usage: 328650kB"
> "
>
> Buffers: shared hit=5047235 read=974471 dirtied=780"
> "
>
> -> Hash Right Join (cost=7694366.79..9691905.74 rows=886647 width=131)
> (actual time=1710903.369..1834807.221 rows=3362294 loops=1)"
> "
>
> Output: ba.bus_ent_inst_id_auto, ba.att_value_1, ba.env_id,
> ba.att_value_num_1, att_dir_hecho.ent_inst_att_str_value,
> att_loc_hecho.ent_inst_att_str_value, att_call (...)"
> "
>
> Hash Cond: ((att_bis.env_id = ba.env_id) AND (att_bis.bus_ent_inst_id =
> ba.bus_ent_inst_id_auto))"
> "
>
> Buffers: shared hit=5047235 read=974471 dirtied=780"
> "
>
> -> Index Only Scan using ix_bus_ent_inst_attr_03 on
> public.bus_ent_inst_attribute att_bis (cost=0.83..1932476.93 rows=8508077
> width=24) (actual time=6.488..116892 (...)"
> "
>
> Output: att_bis.att_id, att_bis.ent_inst_att_str_value, att_bis.env_id,
> att_bis.bus_ent_inst_id, att_bis.reg_status"
> "
>
> Index Cond: (att_bis.att_id = 1117)"
> "
>
> Heap Fetches: 228123"
> "
>
> Buffers: shared hit=218185 read=52064 dirtied=27"
> "
>
> -> Hash (cost=7681066.26..7681066.26 rows=886647 width=115) (actual
> time=1710893.007..1710893.007 rows=3362294 loops=1)"
> "
>
> Output: ba.bus_ent_inst_id_auto, ba.att_value_1, ba.env_id,
> ba.att_value_num_1, att_dir_hecho.ent_inst_att_str_value,
> att_loc_hecho.ent_inst_att_str_value, at (...)"
> "
>
> Buckets: 4194304 (originally 1048576) Batches: 1 (originally 1) Memory
> Usage: 309513kB"
> "
>
> Buffers: shared hit=4829050 read=922407 dirtied=753"
> "
>
> -> Hash Right Join (cost=5969990.07..7681066.26 rows=886647 width=115)
> (actual time=1566042.427..1708291.649 rows=3362294 loops=1)"
> "
>
> Output: ba.bus_ent_inst_id_auto, ba.att_value_1, ba.env_id,
> ba.att_value_num_1, att_dir_hecho.ent_inst_att_str_value,
> att_loc_hecho.ent_inst_att_str_val (...)"
> "
>
> Hash Cond: ((att_call.env_id = ba.env_id) AND (att_call.bus_ent_inst_id =
> ba.bus_ent_inst_id_auto))"
> "
>
> Buffers: shared hit=4829050 read=922407 dirtied=753"
> "
>
> -> Index Only Scan using ix_bus_ent_inst_attr_03 on
> public.bus_ent_inst_attribute att_call (cost=0.83..1655345.90 rows=7287794
> width=24) (actual time= (...)"
> "
>
> Output: att_call.att_id, att_call.ent_inst_att_str_value, att_call.env_id,
> att_call.bus_ent_inst_id, att_call.reg_status"
> "
>
> Index Cond: (att_call.att_id = 1119)"
> "
>
> Heap Fetches: 213801"
> "
>
> Buffers: shared hit=1852588 read=60151 dirtied=23"
> "
>
> -> Hash (cost=5956689.54..5956689.54 rows=886647 width=99) (actual
> time=1566015.832..1566015.832 rows=3362294 loops=1)"
> "
>
> Output: ba.bus_ent_inst_id_auto, ba.att_value_1, ba.env_id,
> ba.att_value_num_1, att_dir_hecho.ent_inst_att_str_value,
> att_loc_hecho.ent_inst_att_s (...)"
> "
>
> Buckets: 4194304 (originally 1048576) Batches: 1 (originally 1) Memory
> Usage: 258291kB"
> "
>
> Buffers: shared hit=2976462 read=862256 dirtied=730"
> "
>
> -> Hash Right Join (cost=4253571.63..5956689.54 rows=886647 width=99)
> (actual time=1355922.435..1563760.249 rows=3362294 loops=1)"
> "
>
> Output: ba.bus_ent_inst_id_auto, ba.att_value_1, ba.env_id,
> ba.att_value_num_1, att_dir_hecho.ent_inst_att_str_value,
> att_loc_hecho.ent_inst (...)"
> "
>
> Hash Cond: ((att_dir_hecho.env_id = ba.env_id) AND
> (att_dir_hecho.bus_ent_inst_id = ba.bus_ent_inst_id_auto))"
> "
>
> Buffers: shared hit=2976462 read=862256 dirtied=730"
> "
>
> -> Index Only Scan using ix_bus_ent_inst_attr_03 on
> public.bus_ent_inst_attribute att_dir_hecho (cost=0.83..1647646.84
> rows=7253898 width= (...)"
> "
>
> Output: att_dir_hecho.att_id, att_dir_hecho.ent_inst_att_str_value,
> att_dir_hecho.env_id, att_dir_hecho.bus_ent_inst_id, att_dir_hecho (...)"
> "
>
> Index Cond: (att_dir_hecho.att_id = 1122)"
> "
>
> Heap Fetches: 221189"
> "
>
> Buffers: shared hit=217265 read=76872 dirtied=96"
> "
>
> -> Hash (cost=4240271.10..4240271.10 rows=886647 width=83) (actual
> time=1355910.157..1355910.157 rows=3362294 loops=1)"
> "
>
> Output: ba.bus_ent_inst_id_auto, ba.att_value_1, ba.env_id,
> ba.att_value_num_1, att_loc_hecho.ent_inst_att_str_value, att_nro.ent_inst
> (...)"
> "
>
> Buckets: 4194304 (originally 1048576) Batches: 1 (originally 1) Memory
> Usage: 224784kB"
> "
>
> Buffers: shared hit=2759197 read=785384 dirtied=634"
> "
>
> -> Hash Right Join (cost=2672428.25..4240271.10 rows=886647 width=83)
> (actual time=1097647.410..1353630.001 rows=3362294 loops=1)"
> "
>
> Output: ba.bus_ent_inst_id_auto, ba.att_value_1, ba.env_id,
> ba.att_value_num_1, att_loc_hecho.ent_inst_att_str_value, att_nro.en (...)"
> "
>
> Hash Cond: ((att_loc_hecho.env_id = ba.env_id) AND
> (att_loc_hecho.bus_ent_inst_id = ba.bus_ent_inst_id_auto))"
> "
>
> Buffers: shared hit=2759197 read=785384 dirtied=634"
> "
>
> -> Index Only Scan using ix_bus_ent_inst_attr_03 on
> public.bus_ent_inst_attribute att_loc_hecho (cost=0.83..1516778.41 rows=66
> (...)"
> "
>
> Output: att_loc_hecho.att_id, att_loc_hecho.ent_inst_att_str_value,
> att_loc_hecho.env_id, att_loc_hecho.bus_ent_inst_id, a (...)"
> "
>
> Index Cond: (att_loc_hecho.att_id = 1133)"
> "
>
> Heap Fetches: 218787"
> "
>
> Buffers: shared hit=332968 read=93935 dirtied=115"
> "
>
> -> Hash (cost=2659127.72..2659127.72 rows=886647 width=67) (actual
> time=1097642.027..1097642.027 rows=3362294 loops=1)"
> "
>
> Output: ba.bus_ent_inst_id_auto, ba.att_value_1, ba.env_id,
> ba.att_value_num_1, att_nro.ent_inst_att_str_value, att_pad.en (...)"
> "
>
> Buckets: 4194304 (originally 1048576) Batches: 1 (originally 1) Memory
> Usage: 215839kB"
> "
>
> Buffers: shared hit=2426229 read=691449 dirtied=519"
> "
>
> -> Hash Right Join (cost=1353880.71..2659127.72 rows=886647 width=67)
> (actual time=466534.722..1095259.942 rows=3362294 (...)"
> "
>
> Output: ba.bus_ent_inst_id_auto, ba.att_value_1, ba.env_id,
> ba.att_value_num_1, att_nro.ent_inst_att_str_value, att_ (...)"
> "
>
> Hash Cond: ((att_nro.env_id = ba.env_id) AND (att_nro.bus_ent_inst_id =
> ba.bus_ent_inst_id_auto))"
> "
>
> Buffers: shared hit=2426229 read=691449 dirtied=519"
> "
>
> -> Index Only Scan using ix_bus_ent_inst_attr_03 on
> public.bus_ent_inst_attribute att_nro (cost=0.83..1262736.66 r (...)"
> "
>
> Output: att_nro.att_id, att_nro.ent_inst_att_str_value, att_nro.env_id,
> att_nro.bus_ent_inst_id, att_nro.reg_s (...)"
> "
>
> Index Cond: (att_nro.att_id = 1135)"
> "
>
> Heap Fetches: 156988"
> "
>
> Buffers: shared hit=1568458 read=151792 dirtied=285"
> "
>
> -> Hash (cost=1340580.18..1340580.18 rows=886647 width=51) (actual
> time=466528.985..466528.985 rows=3362294 loops= (...)"
> "
>
> Output: ba.bus_ent_inst_id_auto, ba.att_value_1, ba.env_id,
> ba.att_value_num_1, att_pad.ent_inst_att_str_value (...)"
> "
>
> Buckets: 4194304 (originally 1048576) Batches: 1 (originally 1) Memory
> Usage: 204115kB"
> "
>
> Buffers: shared hit=857771 read=539657 dirtied=234"
> "
>
> -> Hash Right Join (cost=1265450.85..1340580.18 rows=886647 width=51)
> (actual time=464578.744..465343.707 ro (...)"
> "
>
>
> Output: ba.bus_ent_inst_id_auto, ba.att_value_1, ba.env_id,
> ba.att_value_num_1, att_pad.ent_inst_att_str (...)"
> "
>
>
> Hash Cond: ((att_manz.env_id = ba.env_id) AND (att_manz.bus_ent_inst_id =
> ba.bus_ent_inst_id_auto))"
> "
>
>
> Buffers: shared hit=857771 read=539657 dirtied=234"
> "
>
>
> -> Index Only Scan using ix_bus_ent_inst_attr_03 on
> public.bus_ent_inst_attribute att_manz (cost=0.83. (...)"
> "
>
>
> Output: att_manz.att_id, att_manz.ent_inst_att_str_value, att_manz.env_id,
> att_manz.bus_ent_inst_i (...)"
> "
>
>
> Index Cond: (att_manz.att_id = 1134)"
> "
>
>
> Heap Fetches: 14"
> "
>
>
> Buffers: shared hit=276 read=15"
> "
>
>
> -> Hash (cost=1252150.32..1252150.32 rows=886647 width=35) (actual
> time=464569.271..464569.271 rows=33 (...)"
> "
>
>
> Output: ba.bus_ent_inst_id_auto, ba.att_value_1, ba.env_id,
> ba.att_value_num_1, att_pad.ent_inst_a (...)"
> "
>
>
> Buckets: 4194304 (originally 1048576) Batches: 1 (originally 1) Memory
> Usage: 204114kB"
> "
>
>
> Buffers: shared hit=857495 read=539642 dirtied=234"
> "
>
>
> -> Hash Right Join (cost=1177020.99..1252150.32 rows=886647 width=35)
> (actual time=184587.973..4 (...)"
> "
>
>
> Output: ba.bus_ent_inst_id_auto, ba.att_value_1, ba.env_id,
> ba.att_value_num_1, att_pad.ent_ (...)"
> "
>
>
> Hash Cond: ((att_pad.env_id = ba.env_id) AND (att_pad.bus_ent_inst_id =
> ba.bus_ent_inst_id_a (...)"
> "
>
>
> Buffers: shared hit=857495 read=539642 dirtied=234"
> "
>
>
> -> Index Only Scan using ix_bus_ent_inst_attr_03 on
> public.bus_ent_inst_attribute att_pad (...)"
> "
>
>
> Output: att_pad.att_id, att_pad.ent_inst_att_str_value, att_pad.env_id,
> att_pad.bus_en (...)"
> "
>
>
> Index Cond: (att_pad.att_id = 1136)"
> "
>
>
> Heap Fetches: 54024"
> "
>
>
> Buffers: shared hit=334762 read=60835 dirtied=136"
> "
>
>
> -> Hash (cost=1163720.45..1163720.45 rows=886647 width=19) (actual
> time=184573.023..184573 (...)"
> "
>
>
> Output: ba.bus_ent_inst_id_auto, ba.att_value_1, ba.env_id,
> ba.att_value_num_1"
> "
>
>
> Buckets: 4194304 (originally 1048576) Batches: 1 (originally 1) Memory
> Usage: 200216 (...)"
> "
>
>
> Buffers: shared hit=522733 read=478807 dirtied=98"
> "
>
>
> -> Bitmap Heap Scan on public.bus_ent_instance ba
> (cost=35242.83..1163720.45 rows=88 (...)"
> "
>
>
> Output: ba.bus_ent_inst_id_auto, ba.att_value_1, ba.env_id,
> ba.att_value_num_1"
> "
>
>
> Recheck Cond: ((((ba.bus_ent_inst_name_pre)::text = 'FOTPER'::text) AND
> ((ba.att (...)"
> "
>
>
> Heap Blocks: exact=981056"
> "
>
>
> Buffers: shared hit=522733 read=478807 dirtied=98"
> "
>
>
> -> BitmapOr (cost=35242.83..35242.83 rows=896239 width=0) (actual
> time=33401.6 (...)"
> "
>
>
> Buffers: shared hit=43 read=20441"
> "
>
>
> -> Bitmap Index Scan on ix_bus_ent_instance_atts_namenum_01 (cost=0.00..
> (...)"
> "
>
>
> Index Cond: (((ba.bus_ent_inst_name_pre)::text = 'FOTPER'::text) AND (...)"
> "
>
>
> Buffers: shared hit=9 read=5030"
> "
>
>
> -> Bitmap Index Scan on ix_bus_ent_instance_atts_namenum_01 (cost=0.00..
> (...)"
> "
>
>
> Index Cond: (((ba.bus_ent_inst_name_pre)::text = 'FOTPER'::text) AND (...)"
> "
>
>
> Buffers: shared hit=34 read=15411"
> " -> Sort
> (cost=125485.07..125573.87 rows=35520 width=13) (actual time=5.831..312.164
> rows=3217523 loops=1)"
> " Output:
> depto2.att_value_1, depto2.env_id, depto2.bus_ent_inst_name_num"
> " Sort Key:
> depto2.env_id, depto2.bus_ent_inst_name_num"
> " Sort Method:
> quicksort Memory: 26kB"
> " Buffers:
> shared hit=7 read=2"
> " -> Bitmap
> Heap Scan on public.bus_ent_instance depto2 (cost=971.85..122800.41
> rows=35520 width=13) (actual time=5.758..5.776 rows=21 loops=1)"
> "
> Output:
> depto2.att_value_1, depto2.env_id, depto2.bus_ent_inst_name_num"
> "
> Recheck
> Cond: (depto2.bus_ent_id = 1091)"
> " Heap
> Blocks: exact=5"
> "
> Buffers: shared hit=7 read=2"
> " ->
> Bitmap Index Scan on ix_bus_ent_instance_01 (cost=0.00..962.97 rows=35520
> width=0) (actual time=5.733..5.733 rows=21 loops=1)"
> "
> Index Cond: (depto2.bus_ent_id = 1091)"
> "
> Buffers: shared hit=2 read=2"
> " -> Sort
> (cost=125485.07..125573.87 rows=35520 width=13) (actual
> time=14.418..320.637
> rows=3217335 loops=1)"
> " Output:
> loc2.att_value_1,
> loc2.env_id, loc2.bus_ent_inst_name_num"
> " Sort Key: loc2.env_id,
> loc2.bus_ent_inst_name_num"
> " Sort Method: quicksort
> Memory: 76kB"
> " Buffers: shared hit=112
> read=5"
> " -> Bitmap Heap Scan on
> public.bus_ent_instance loc2 (cost=971.85..122800.41 rows=35520 width=13)
> (actual time=13.305..13.922 rows=725 loops=1)"
> " Output:
> loc2.att_value_1, loc2.env_id, loc2.bus_ent_inst_name_num"
> " Recheck Cond:
> (loc2.bus_ent_id = 1165)"
> " Heap Blocks:
> exact=110"
> " Buffers: shared
> hit=112 read=5"
> " -> Bitmap Index
> Scan on ix_bus_ent_instance_01 (cost=0.00..962.97 rows=35520 width=0)
> (actual time=13.262..13.262 rows=725 loops=1)"
> " Index Cond:
> (loc2.bus_ent_id = 1165)"
> " Buffers:
> shared hit=2 read=5"
> " -> Hash (cost=4.35..4.35 rows=235 width=552) (actual
> time=0.175..0.175 rows=235 loops=1)"
> " Output: bi.bus_ent_inst_name_num, bi.att_value_num_7,
> bi.att_value_10"
> " Buckets: 1024 Batches: 1 Memory Usage: 19kB"
> " Buffers: local hit=2"
> " -> Seq Scan on pg_temp_179.temp_table bi
> (cost=0.00..4.35 rows=235 width=552) (actual time=0.015..0.055 rows=235
> loops=1)"
> " Output: bi.bus_ent_inst_name_num,
> bi.att_value_num_7, bi.att_value_10"
> " Buffers: local hit=2"
> " -> Index Scan using ix_bus_ent_inst_attr_01 on
> public.bus_ent_inst_attribute a2 (cost=0.58..237.03 rows=123 width=20)
> (actual time=23.167..23.168 rows=1 loops=306)"
> " Output: a2.env_id, a2.bus_ent_inst_id, a2.att_id,
> a2.att_row_id_auto, a2.att_index_id, a2.ent_inst_att_num_value,
> a2.ent_inst_att_str_value, a2.ent_inst_att_dte_value,
> a2.ent_inst_att_doc_id, a2.ent_inst_att_tran_1, a2.ent_inst_att_tran_2, a2
> (...)"
> " Index Cond: ((a2.bus_ent_inst_id = ba.bus_ent_inst_id_auto)
> AND (a2.att_id = 1083))"
> " Buffers: shared hit=635 read=895"
> " SubPlan 1"
> " -> Index Scan using ix_bus_ent_inst_attr_01 on
> public.bus_ent_inst_attribute a (cost=0.58..141.91 rows=72 width=16)
> (actual time=0.646..0.647 rows=1 loops=306)"
> " Output: a.ent_inst_att_str_value"
> " Index Cond: ((ba.bus_ent_inst_id_auto = a.bus_ent_inst_id)
> AND (a.att_id = 1071))"
> " Filter: (a.reg_status = 0)"
> " Buffers: shared hit=1434 read=31"
> " SubPlan 2"
> " -> Index Scan using ix_bus_ent_inst_attr_01 on
> public.bus_ent_inst_attribute t (cost=0.58..46.15 rows=21 width=16)
> (actual
> time=0.839..0.841 rows=0 loops=306)"
> " Output: t.ent_inst_att_str_value"
> " Index Cond: ((ba.bus_ent_inst_id_auto = t.bus_ent_inst_id)
> AND (t.att_id = 1141))"
> " Filter: (t.reg_status = 0)"
> " Buffers: shared hit=1217 read=42"
> "Planning time: 18.329 ms"
> "Execution time: 1972336.524 ms"
>
>
>
> --
> Sent from:
> https://www.postgresql-archive.org/PostgreSQL-performance-f2050081.html
>
>
>
^ permalink raw reply [nested|flat] 112+ messages in thread
* Slow Query
@ 2020-10-14 17:30 Parth Shah <parth@polimorphic.com>
0 siblings, 1 reply; 112+ messages in thread
From: Parth Shah @ 2020-10-14 17:30 UTC (permalink / raw)
To: pgsql-performance; +Cc: Daniel Smith <daniel@polimorphic.com>
Hi all,
We've been struggling with a slow query! -- and it's been exploding as rows
have been added to relevant tables. It seems like a fairly common workflow,
so we think we're overlooking the potential for an index or rewriting the
query.
I've linked a document compiling the information as per the Postgresql
recommendation for Slow Query Questions. Here's the link:
https://docs.google.com/document/d/10qO5jkQNVtKw2Af1gcKAKiNw7tYFNQruzOQrUYXd4hk/edit?usp=sharing
(we've enabled commenting)
Here's a high-level summary of the issue:
______
We’re trying to show a list of active conversations. Each conversation
(named a spool in the database) has multiple threads, kind of like Slack
channels. And the messages are stored in each thread. We want to return the
30 most recent conversations with recency determined as the most recent
message in any thread of the conversation you are a participant of (you may
not be a participant of certain threads in a conversation so it’s important
those don’t leak sensitive data).
We found that as the number of threads increases, the query slowed down
dramatically. We think the issue has to do with the fact that there is no
easy way to go from a thread you are a participant to its most recent
message, however, it is possible the issue is elsewhere. We’ve provided the
full query and a simplified query of where we think the issue is, along
with the EXPLAIN ANALYZE BUFFERS result. We figure this is not exactly an
uncommon use case, so it’s likely that we are overlooking the potential for
some missing indices or a better way to write the query. We appreciate the
help and any advice!
______
We'd really appreciate any help and advice!
Best,
Parth
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Slow Query
@ 2020-10-14 19:17 Michael Lewis <mlewis@entrata.com>
parent: Parth Shah <parth@polimorphic.com>
0 siblings, 1 reply; 112+ messages in thread
From: Michael Lewis @ 2020-10-14 19:17 UTC (permalink / raw)
To: Parth Shah <parth@polimorphic.com>; +Cc: pgsql-performance; Daniel Smith <daniel@polimorphic.com>
Is there no index on thread.spool? What about notification.user? How about
message.time (without thread as a leading column). Those would all seem
very significant. Your row counts are very low to have a query perform so
badly. Work_mem could probably be increased above 4MB, but it isn't hurting
this query in particular.
My primary concern is that the query is rather chaotic at a glance. It
would be great to re-write and remove the unneeded keywords, double quotes,
totally worthless parentheses, etc. Something like the below may help you
see the crux of the query and what could be done and understand how many
rows might be coming out of those subqueries. I re-ordered some joins and
there might be syntax errors, but give it a shot once you've added the
indexes suggested above.
SELECT
spool.id,
handle.handle,
spool.name,
thread.id,
case.closed,
notification.read,
notification2.time,
message.message,
message.time,
message.author,
thread.name,
location.geo
FROM
spool
JOIN handle ON handle.id = spool.id
JOIN thread ON thread.spool = spool.id
JOIN message ON message.thread = thread.id
LEFT JOIN location ON location.id = spool.location
LEFT JOIN case ON case.id = spool.id
LEFT JOIN notification ON notification.user =
'b16690e4-a3c5-4868-945e-c2458c27a525'
AND
notification.id = (
SELECT
notification3.id
FROM
notification AS notification3
JOIN notification_thread ON notification_thread.id = notification3.id
JOIN thread AS thread2 ON thread2.id = notification_thread.thread
WHERE
thread2.spool = spool.id
AND notification3.user = 'b16690e4-a3c5-4868-945e-c2458c27a525'
AND notification3.time <= '2020-09-30 16:32:38.054558'
ORDER BY
notification3.time DESC
LIMIT 1
)
LEFT JOIN notification AS notification2 ON notification2.user =
'b16690e4-a3c5-4868-945e-c2458c27a525'
AND notification2.id = (
SELECT
notification3.id
FROM
notification AS notification3
JOIN notification_thread ON notification_thread.id = notification3.id
JOIN thread AS thread2 ON thread2.id = notification_thread.thread
WHERE
thread2.spool = spool.id
AND notification3.user = 'b16690e4-a3c5-4868-945e-c2458c27a525'
AND notification3.time > '2020-09-30 16:32:38.054558'
ORDER BY
notification3.time DESC
LIMIT 1
)
WHERE
message.time = (
SELECT
MAX ( message2.time )
FROM
message AS message2
JOIN thread AS thread2 ON thread2.id = message2.thread
JOIN participant ON participant.thread = thread2.id
JOIN identity ON identity.id = participant.identity
LEFT JOIN relation ON relation.to = identity.id
AND relation.from = 'b16690e4-a3c5-4868-945e-c2458c27a525'
AND relation.manages = TRUE
WHERE
NOT message2.draft
AND ( identity.id = 'b16690e4-a3c5-4868-945e-c2458c27a525' OR NOT
relation.to IS NULL )
AND thread2.spool = spool.id
LIMIT 1
)
AND notification.id IS NOT NULL
ORDER BY
message.time DESC
LIMIT 31;
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Slow Query
@ 2020-10-15 01:11 Parth Shah <parth@polimorphic.com>
parent: Michael Lewis <mlewis@entrata.com>
0 siblings, 1 reply; 112+ messages in thread
From: Parth Shah @ 2020-10-15 01:11 UTC (permalink / raw)
To: Michael Lewis <mlewis@entrata.com>; +Cc: pgsql-performance; Daniel Smith <daniel@polimorphic.com>
Hi all,
Thanks, Michael (and Martin other thread)! We added those indexes you
suggested, and went ahead and added indexes for all our foreign keys. We
also added one combination index on notification (user, time). It led to a
small constant factor speed up (2x) but is still taking a 13+ seconds. :(
Still seems aggressively bad.
I've attached the updated, cleaned up query and explain analyze result (the
extra chaos was due to the fact that we're using
https://hackage.haskell.org/package/esqueleto-3.2.3/docs/Database-Esqueleto.html
to
generate the SQL). Maybe we're missing some multi-column indexes?
Best,
Parth
On Wed, Oct 14, 2020 at 3:18 PM Michael Lewis <mlewis@entrata.com> wrote:
> Is there no index on thread.spool? What about notification.user? How about
> message.time (without thread as a leading column). Those would all seem
> very significant. Your row counts are very low to have a query perform so
> badly. Work_mem could probably be increased above 4MB, but it isn't hurting
> this query in particular.
>
> My primary concern is that the query is rather chaotic at a glance. It
> would be great to re-write and remove the unneeded keywords, double quotes,
> totally worthless parentheses, etc. Something like the below may help you
> see the crux of the query and what could be done and understand how many
> rows might be coming out of those subqueries. I re-ordered some joins and
> there might be syntax errors, but give it a shot once you've added the
> indexes suggested above.
>
> SELECT
>
> spool.id,
>
> handle.handle,
>
> spool.name,
>
> thread.id,
>
> case.closed,
>
> notification.read,
>
> notification2.time,
>
> message.message,
>
> message.time,
>
> message.author,
>
> thread.name,
>
> location.geo
>
> FROM
>
> spool
>
> JOIN handle ON handle.id = spool.id
>
> JOIN thread ON thread.spool = spool.id
>
> JOIN message ON message.thread = thread.id
>
> LEFT JOIN location ON location.id = spool.location
>
> LEFT JOIN case ON case.id = spool.id
>
> LEFT JOIN notification ON notification.user =
> 'b16690e4-a3c5-4868-945e-c2458c27a525'
>
> AND
>
> notification.id = (
>
> SELECT
>
> notification3.id
>
> FROM
>
> notification AS notification3
>
> JOIN notification_thread ON notification_thread.id = notification3.id
>
> JOIN thread AS thread2 ON thread2.id = notification_thread.thread
>
> WHERE
>
> thread2.spool = spool.id
>
> AND notification3.user = 'b16690e4-a3c5-4868-945e-c2458c27a525'
>
> AND notification3.time <= '2020-09-30 16:32:38.054558'
>
> ORDER BY
>
> notification3.time DESC
>
> LIMIT 1
>
> )
>
> LEFT JOIN notification AS notification2 ON notification2.user =
> 'b16690e4-a3c5-4868-945e-c2458c27a525'
>
> AND notification2.id = (
>
> SELECT
>
> notification3.id
>
> FROM
>
> notification AS notification3
>
> JOIN notification_thread ON notification_thread.id = notification3.id
>
> JOIN thread AS thread2 ON thread2.id = notification_thread.thread
>
> WHERE
>
> thread2.spool = spool.id
>
> AND notification3.user = 'b16690e4-a3c5-4868-945e-c2458c27a525'
>
> AND notification3.time > '2020-09-30 16:32:38.054558'
>
> ORDER BY
>
> notification3.time DESC
>
> LIMIT 1
>
> )
>
> WHERE
>
> message.time = (
>
> SELECT
>
> MAX ( message2.time )
>
> FROM
>
> message AS message2
>
> JOIN thread AS thread2 ON thread2.id = message2.thread
>
> JOIN participant ON participant.thread = thread2.id
>
> JOIN identity ON identity.id = participant.identity
>
> LEFT JOIN relation ON relation.to = identity.id
>
> AND relation.from = 'b16690e4-a3c5-4868-945e-c2458c27a525'
>
> AND relation.manages = TRUE
>
> WHERE
>
> NOT message2.draft
>
> AND ( identity.id = 'b16690e4-a3c5-4868-945e-c2458c27a525' OR NOT
> relation.to IS NULL )
>
> AND thread2.spool = spool.id
>
> LIMIT 1
>
> )
>
> AND notification.id IS NOT NULL
>
> ORDER BY
>
> message.time DESC
>
> LIMIT 31;
>
Attachments:
[application/octet-stream] inbox.sql (2.2K, ../../CA++17FKqvU2p7RN996dcXE69tsKxyPsMRd3FuSo1ivxX87RmBw@mail.gmail.com/3-inbox.sql)
download
[application/json] explain.json (28.5K, ../../CA++17FKqvU2p7RN996dcXE69tsKxyPsMRd3FuSo1ivxX87RmBw@mail.gmail.com/4-explain.json)
download
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Slow Query
@ 2020-10-15 04:51 Michael Lewis <mlewis@entrata.com>
parent: Parth Shah <parth@polimorphic.com>
0 siblings, 0 replies; 112+ messages in thread
From: Michael Lewis @ 2020-10-15 04:51 UTC (permalink / raw)
To: Parth Shah <parth@polimorphic.com>; +Cc: pgsql-performance; Daniel Smith <daniel@polimorphic.com>
Based on the execution plan, it looks like the part that takes 13 seconds
of the total 14.4 seconds is just calculating the max time used in the
where clause. Anytime I see an OR involved in a plan gone off the rails, I
always always check if re-writing the query some other way may be faster.
How's the plan for something like this?
WHERE message.time = greatest( *sub1.time*, *sub2.time* )
/* sub1.time */
(
select
MAX ( message2.time )
FROM
message AS message2
JOIN thread AS thread2 ON thread2.id = message2.thread
JOIN participant ON participant.thread = thread2.id
WHERE
NOT message2.draft
AND participant.identity = 'b16690e4-a3c5-4868-945e-c2458c27a525'
AND thread2.spool = spool.id
)
/* sub2.time */
(
select
MAX ( message2.time )
FROM
message AS message2
JOIN thread AS thread2 ON thread2.id = message2.thread
JOIN participant ON participant.thread = thread2.id
JOIN relation ON relation.to = participant.identity
AND relation.from = 'b16690e4-a3c5-4868-945e-c2458c27a525'
AND relation.manages = TRUE
WHERE
NOT message2.draft
AND thread2.spool = spool.id
)
>
^ permalink raw reply [nested|flat] 112+ messages in thread
* slow query
@ 2021-06-08 16:03 Ayub Khan <ayub.hp@gmail.com>
0 siblings, 5 replies; 112+ messages in thread
From: Ayub Khan @ 2021-06-08 16:03 UTC (permalink / raw)
To: Pgsql Performance <pgsql-performance@lists.postgresql.org>
I checked all the indexes are defined on the tables however the query seems
slow, below is the plan. Can any one give any pointers to verify ?
SELECT a.menu_item_id, a.menu_item_name, a.menu_item_category_id,
b.menu_item_category_desc, c.menu_item_variant_id,
c.menu_item_variant_type_id, c.price, c.size_id,
c.parent_menu_item_variant_id, d.menu_item_variant_type_desc,
e.size_desc, f.currency_code, a.image, a.mark_id, m.mark_name
FROM menu_item_category AS b, menu_item_variant AS c,
menu_item_variant_type AS d, item_size AS e, restaurant AS f,
menu_item AS a
LEFT OUTER JOIN mark AS m ON (a.mark_id = m.mark_id) WHERE
a.menu_item_category_id = b.menu_item_category_id AND a.menu_item_id =
c.menu_item_id AND c.menu_item_variant_type_id =
d.menu_item_variant_type_id AND d.is_hidden = 'false' AND c.size_id =
e.size_id AND a.restaurant_id = f.restaurant_id AND f.restaurant_id =
1528 AND (a.menu_item_category_id = NULL OR NULL IS NULL)
AND c.menu_item_variant_id = (SELECT min(menu_item_variant_id) FROM
menu_item_variant WHERE menu_item_id = a.menu_item_id AND deleted =
'N' limit 1) AND a.active = 'Y'
AND (CONCAT_WS('', ',', a.hidden_branch_ids, ',') NOT LIKE
CONCAT_WS('', '%,4191,%') OR NULL IS NULL)
AND is_menu_item_available(a.menu_item_id, 'Y') = 'Y'
ORDER BY a.row_order, menu_item_id;
below is the plan
Sort (cost=189.27..189.27 rows=1 width=152) (actual time=5.876..5.885
rows=89 loops=1)
" Sort Key: a.row_order, a.menu_item_id"
Sort Method: quicksort Memory: 48kB
-> Nested Loop Left Join (cost=5.19..189.26 rows=1 width=152)
(actual time=0.188..5.809 rows=89 loops=1)
Join Filter: (a.mark_id = m.mark_id)
Rows Removed by Join Filter: 267
-> Nested Loop (cost=5.19..188.19 rows=1 width=148) (actual
time=0.181..5.629 rows=89 loops=1)
-> Nested Loop (cost=4.90..185.88 rows=1 width=152)
(actual time=0.174..5.443 rows=89 loops=1)
-> Nested Loop (cost=4.61..185.57 rows=1
width=144) (actual time=0.168..5.272 rows=89 loops=1)
-> Nested Loop (cost=4.32..185.25 rows=1
width=136) (actual time=0.162..5.066 rows=89 loops=1)
-> Nested Loop (cost=0.71..179.62
rows=1 width=99) (actual time=0.137..3.986 rows=89 loops=1)
-> Index Scan using
menu_item_restaurant_id on menu_item a (cost=0.42..177.31 rows=1
width=87) (actual time=0.130..3.769 rows=89 loops=1)
Index Cond: (restaurant_id = 1528)
" Filter: ((active =
'Y'::bpchar) AND (is_menu_item_available(menu_item_id, 'Y'::bpchar) =
'Y'::bpchar))"
Rows Removed by Filter: 194
-> Index Scan using
menu_item_category_pk on menu_item_category b (cost=0.29..2.31 rows=1
width=20) (actual time=0.002..0.002 rows=1 loops=89)
Index Cond:
(menu_item_category_id = a.menu_item_category_id)
-> Index Scan using
menu_item_variant_pk on menu_item_variant c (cost=3.60..5.62 rows=1
width=45) (actual time=0.002..0.002 rows=1 loops=89)
Index Cond:
(menu_item_variant_id = (SubPlan 1))
Filter: (a.menu_item_id = menu_item_id)
SubPlan 1
-> Limit (cost=3.17..3.18
rows=1 width=8) (actual time=0.009..0.009 rows=1 loops=89)
-> Aggregate
(cost=3.17..3.18 rows=1 width=8) (actual time=0.008..0.008 rows=1
loops=89)
-> Index Scan
using "idx$$_023a0001" on menu_item_variant (cost=0.43..3.15 rows=8
width=8) (actual time=0.004..0.007 rows=7 loops=89)
Index Cond:
(menu_item_id = a.menu_item_id)
Filter:
(deleted = 'N'::bpchar)
Rows Removed
by Filter: 4
-> Index Scan using
menu_item_variant_type_pk on menu_item_variant_type d
(cost=0.29..0.31 rows=1 width=16) (actual time=0.002..0.002 rows=1
loops=89)
Index Cond: (menu_item_variant_type_id
= c.menu_item_variant_type_id)
Filter: ((is_hidden)::text = 'false'::text)
-> Index Scan using size_pk on item_size e
(cost=0.29..0.31 rows=1 width=16) (actual time=0.001..0.001 rows=1
loops=89)
Index Cond: (size_id = c.size_id)
-> Index Scan using "restaurant_idx$$_274b003d" on
restaurant f (cost=0.29..2.30 rows=1 width=12) (actual
time=0.001..0.002 rows=1 loops=89)
Index Cond: (restaurant_id = 1528)
-> Seq Scan on mark m (cost=0.00..1.03 rows=3 width=12)
(actual time=0.000..0.001 rows=3 loops=89)
Planning Time: 1.510 ms
Execution Time: 5.972 ms
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: slow query
@ 2021-06-08 16:08 Christophe Pettus <xof@thebuild.com>
parent: Ayub Khan <ayub.hp@gmail.com>
4 siblings, 0 replies; 112+ messages in thread
From: Christophe Pettus @ 2021-06-08 16:08 UTC (permalink / raw)
To: Ayub Khan <ayub.hp@gmail.com>; +Cc: Pgsql Performance <pgsql-performance@lists.postgresql.org>
> On Jun 8, 2021, at 09:03, Ayub Khan <ayub.hp@gmail.com> wrote:
> I checked all the indexes are defined on the tables however the query seems slow, below is the plan.
It's currently running in slightly under six milliseconds. That seems reasonably fast given the number of operations required to fulfill it.
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: slow query
@ 2021-06-08 16:32 Ayub Khan <ayub.hp@gmail.com>
parent: Ayub Khan <ayub.hp@gmail.com>
4 siblings, 1 reply; 112+ messages in thread
From: Ayub Khan @ 2021-06-08 16:32 UTC (permalink / raw)
To: Pgsql Performance <pgsql-performance@lists.postgresql.org>
In AWS RDS performance insights the client writes is high and the api
which receives data on the mobile side is slow during load test.
On Tue, 8 Jun 2021, 19:03 Ayub Khan, <ayub.hp@gmail.com> wrote:
>
> I checked all the indexes are defined on the tables however the query
> seems slow, below is the plan. Can any one give any pointers to verify ?
>
> SELECT a.menu_item_id, a.menu_item_name, a.menu_item_category_id, b.menu_item_category_desc, c.menu_item_variant_id, c.menu_item_variant_type_id, c.price, c.size_id, c.parent_menu_item_variant_id, d.menu_item_variant_type_desc, e.size_desc, f.currency_code, a.image, a.mark_id, m.mark_name
>
> FROM menu_item_category AS b, menu_item_variant AS c, menu_item_variant_type AS d, item_size AS e, restaurant AS f, menu_item AS a
>
> LEFT OUTER JOIN mark AS m ON (a.mark_id = m.mark_id) WHERE a.menu_item_category_id = b.menu_item_category_id AND a.menu_item_id = c.menu_item_id AND c.menu_item_variant_type_id = d.menu_item_variant_type_id AND d.is_hidden = 'false' AND c.size_id = e.size_id AND a.restaurant_id = f.restaurant_id AND f.restaurant_id = 1528 AND (a.menu_item_category_id = NULL OR NULL IS NULL)
>
> AND c.menu_item_variant_id = (SELECT min(menu_item_variant_id) FROM menu_item_variant WHERE menu_item_id = a.menu_item_id AND deleted = 'N' limit 1) AND a.active = 'Y'
> AND (CONCAT_WS('', ',', a.hidden_branch_ids, ',') NOT LIKE CONCAT_WS('', '%,4191,%') OR NULL IS NULL)
> AND is_menu_item_available(a.menu_item_id, 'Y') = 'Y'
>
> ORDER BY a.row_order, menu_item_id;
>
>
> below is the plan
>
>
> Sort (cost=189.27..189.27 rows=1 width=152) (actual time=5.876..5.885 rows=89 loops=1)
> " Sort Key: a.row_order, a.menu_item_id"
> Sort Method: quicksort Memory: 48kB
> -> Nested Loop Left Join (cost=5.19..189.26 rows=1 width=152) (actual time=0.188..5.809 rows=89 loops=1)
> Join Filter: (a.mark_id = m.mark_id)
> Rows Removed by Join Filter: 267
> -> Nested Loop (cost=5.19..188.19 rows=1 width=148) (actual time=0.181..5.629 rows=89 loops=1)
> -> Nested Loop (cost=4.90..185.88 rows=1 width=152) (actual time=0.174..5.443 rows=89 loops=1)
> -> Nested Loop (cost=4.61..185.57 rows=1 width=144) (actual time=0.168..5.272 rows=89 loops=1)
> -> Nested Loop (cost=4.32..185.25 rows=1 width=136) (actual time=0.162..5.066 rows=89 loops=1)
> -> Nested Loop (cost=0.71..179.62 rows=1 width=99) (actual time=0.137..3.986 rows=89 loops=1)
> -> Index Scan using menu_item_restaurant_id on menu_item a (cost=0.42..177.31 rows=1 width=87) (actual time=0.130..3.769 rows=89 loops=1)
> Index Cond: (restaurant_id = 1528)
> " Filter: ((active = 'Y'::bpchar) AND (is_menu_item_available(menu_item_id, 'Y'::bpchar) = 'Y'::bpchar))"
> Rows Removed by Filter: 194
> -> Index Scan using menu_item_category_pk on menu_item_category b (cost=0.29..2.31 rows=1 width=20) (actual time=0.002..0.002 rows=1 loops=89)
> Index Cond: (menu_item_category_id = a.menu_item_category_id)
> -> Index Scan using menu_item_variant_pk on menu_item_variant c (cost=3.60..5.62 rows=1 width=45) (actual time=0.002..0.002 rows=1 loops=89)
> Index Cond: (menu_item_variant_id = (SubPlan 1))
> Filter: (a.menu_item_id = menu_item_id)
> SubPlan 1
> -> Limit (cost=3.17..3.18 rows=1 width=8) (actual time=0.009..0.009 rows=1 loops=89)
> -> Aggregate (cost=3.17..3.18 rows=1 width=8) (actual time=0.008..0.008 rows=1 loops=89)
> -> Index Scan using "idx$$_023a0001" on menu_item_variant (cost=0.43..3.15 rows=8 width=8) (actual time=0.004..0.007 rows=7 loops=89)
> Index Cond: (menu_item_id = a.menu_item_id)
> Filter: (deleted = 'N'::bpchar)
> Rows Removed by Filter: 4
> -> Index Scan using menu_item_variant_type_pk on menu_item_variant_type d (cost=0.29..0.31 rows=1 width=16) (actual time=0.002..0.002 rows=1 loops=89)
> Index Cond: (menu_item_variant_type_id = c.menu_item_variant_type_id)
> Filter: ((is_hidden)::text = 'false'::text)
> -> Index Scan using size_pk on item_size e (cost=0.29..0.31 rows=1 width=16) (actual time=0.001..0.001 rows=1 loops=89)
> Index Cond: (size_id = c.size_id)
> -> Index Scan using "restaurant_idx$$_274b003d" on restaurant f (cost=0.29..2.30 rows=1 width=12) (actual time=0.001..0.002 rows=1 loops=89)
> Index Cond: (restaurant_id = 1528)
> -> Seq Scan on mark m (cost=0.00..1.03 rows=3 width=12) (actual time=0.000..0.001 rows=3 loops=89)
> Planning Time: 1.510 ms
> Execution Time: 5.972 ms
>
>
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: slow query
@ 2021-06-08 17:14 Tom Lane <tgl@sss.pgh.pa.us>
parent: Ayub Khan <ayub.hp@gmail.com>
4 siblings, 0 replies; 112+ messages in thread
From: Tom Lane @ 2021-06-08 17:14 UTC (permalink / raw)
To: Ayub Khan <ayub.hp@gmail.com>; +Cc: Pgsql Performance <pgsql-performance@lists.postgresql.org>
Ayub Khan <ayub.hp@gmail.com> writes:
> I checked all the indexes are defined on the tables however the query seems
> slow, below is the plan. Can any one give any pointers to verify ?
You might try to do something about the poor selectivity estimate here:
> -> Index Scan using
> menu_item_restaurant_id on menu_item a (cost=0.42..177.31 rows=1
> width=87) (actual time=0.130..3.769 rows=89 loops=1)
> Index Cond: (restaurant_id = 1528)
> " Filter: ((active =
> 'Y'::bpchar) AND (is_menu_item_available(menu_item_id, 'Y'::bpchar) =
> 'Y'::bpchar))"
> Rows Removed by Filter: 194
If the planner realized that this'd produce O(100) rows not 1,
it'd likely have picked a different plan. I'm guessing that
the issue is lack of knowledge about what is_menu_item_available()
will do. Maybe you could replace that with a status column?
regards, tom lane
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: slow query
@ 2021-06-08 17:43 Ayub Khan <ayub.hp@gmail.com>
parent: Ayub Khan <ayub.hp@gmail.com>
4 siblings, 0 replies; 112+ messages in thread
From: Ayub Khan @ 2021-06-08 17:43 UTC (permalink / raw)
To: Pgsql Performance <pgsql-performance@lists.postgresql.org>
below is function definition of is_menu_item_available, for each item
based on current day time it returns when it's available or not. The same
api works fine on oracle, I am seeing this slowness after migrating the
queries to postgresql RDS on AWS
CREATE OR REPLACE FUNCTION is_menu_item_available(
i_menu_item_id bigint,
i_check_availability character)
RETURNS character
LANGUAGE 'plpgsql'
COST 100
VOLATILE PARALLEL UNSAFE
AS $BODY$
DECLARE
l_current_day NUMERIC(1);
o_time CHARACTER VARYING(10);
l_current_interval INTERVAL DAY TO SECOND(2);
item_available_count NUMERIC(10);
BEGIN
item_available_count := 0;
BEGIN
IF i_check_availability = 'Y' THEN
BEGIN
SELECT
CASE TO_CHAR(now(), 'fmday')
WHEN 'monday' THEN 1
WHEN 'tuesday' THEN 2
WHEN 'wednesday' THEN 3
WHEN 'thursday' THEN 4
WHEN 'friday' THEN 5
WHEN 'saturday' THEN 6
WHEN 'sunday' THEN 7
END AS d
INTO STRICT l_current_day;
select (('0 ' ||
EXTRACT (HOUR FROM ((now() at time zone 'UTC') at time zone '+03:00'))
|| ':' ||
EXTRACT (minute FROM ((now() at time zone 'UTC') at time zone
'+03:00')) || ':00') :: interval)
INTO l_current_interval;
END;
BEGIN
SELECT
COUNT(*)
INTO STRICT item_available_count
FROM menu_item_availability
WHERE menu_item_id = i_menu_item_id;
IF item_available_count = 0 THEN
RETURN 'Y';
ELSE
SELECT
COUNT(*)
INTO STRICT item_available_count
FROM menu_item_availability AS mia, availability AS av
WHERE mia.menu_item_id = i_menu_item_id
AND mia.availability_id = av.id
AND date_trunc('DAY',now()) + l_current_interval >= (CASE
WHEN l_current_interval < '6 hour'::INTERVAL THEN
date_trunc('DAY',now()) + av.start_time - (1::NUMERIC || ' days')::INTERVAL
WHEN l_current_interval >= '6 hour'::INTERVAL THEN
date_trunc('DAY',now())+ av.start_time
END) AND date_trunc('DAY',now()) + l_current_interval <= (CASE
WHEN l_current_interval < '6 hour'::INTERVAL THEN
date_trunc('DAY',now()) + av.end_time - (1::NUMERIC || ' days')::INTERVAL
WHEN l_current_interval >= '6 hour'::INTERVAL THEN
date_trunc('DAY',now()) + av.end_time
END) AND (av.day_of_week LIKE CONCAT_WS('', '%', l_current_day, '%')
OR av.day_of_week LIKE '%0%') AND is_deleted = 0;
END IF;
END;
BEGIN
IF item_available_count > 0 THEN
RETURN 'Y';
ELSE
RETURN 'N';
END IF;
END;
ELSE
RETURN 'Y';
END IF;
END;
END;
$BODY$;
On Tue, Jun 8, 2021 at 7:03 PM Ayub Khan <ayub.hp@gmail.com> wrote:
>
> I checked all the indexes are defined on the tables however the query
> seems slow, below is the plan. Can any one give any pointers to verify ?
>
> SELECT a.menu_item_id, a.menu_item_name, a.menu_item_category_id, b.menu_item_category_desc, c.menu_item_variant_id, c.menu_item_variant_type_id, c.price, c.size_id, c.parent_menu_item_variant_id, d.menu_item_variant_type_desc, e.size_desc, f.currency_code, a.image, a.mark_id, m.mark_name
>
> FROM menu_item_category AS b, menu_item_variant AS c, menu_item_variant_type AS d, item_size AS e, restaurant AS f, menu_item AS a
>
> LEFT OUTER JOIN mark AS m ON (a.mark_id = m.mark_id) WHERE a.menu_item_category_id = b.menu_item_category_id AND a.menu_item_id = c.menu_item_id AND c.menu_item_variant_type_id = d.menu_item_variant_type_id AND d.is_hidden = 'false' AND c.size_id = e.size_id AND a.restaurant_id = f.restaurant_id AND f.restaurant_id = 1528 AND (a.menu_item_category_id = NULL OR NULL IS NULL)
>
> AND c.menu_item_variant_id = (SELECT min(menu_item_variant_id) FROM menu_item_variant WHERE menu_item_id = a.menu_item_id AND deleted = 'N' limit 1) AND a.active = 'Y'
> AND (CONCAT_WS('', ',', a.hidden_branch_ids, ',') NOT LIKE CONCAT_WS('', '%,4191,%') OR NULL IS NULL)
> AND is_menu_item_available(a.menu_item_id, 'Y') = 'Y'
>
> ORDER BY a.row_order, menu_item_id;
>
>
> below is the plan
>
>
> Sort (cost=189.27..189.27 rows=1 width=152) (actual time=5.876..5.885 rows=89 loops=1)
> " Sort Key: a.row_order, a.menu_item_id"
> Sort Method: quicksort Memory: 48kB
> -> Nested Loop Left Join (cost=5.19..189.26 rows=1 width=152) (actual time=0.188..5.809 rows=89 loops=1)
> Join Filter: (a.mark_id = m.mark_id)
> Rows Removed by Join Filter: 267
> -> Nested Loop (cost=5.19..188.19 rows=1 width=148) (actual time=0.181..5.629 rows=89 loops=1)
> -> Nested Loop (cost=4.90..185.88 rows=1 width=152) (actual time=0.174..5.443 rows=89 loops=1)
> -> Nested Loop (cost=4.61..185.57 rows=1 width=144) (actual time=0.168..5.272 rows=89 loops=1)
> -> Nested Loop (cost=4.32..185.25 rows=1 width=136) (actual time=0.162..5.066 rows=89 loops=1)
> -> Nested Loop (cost=0.71..179.62 rows=1 width=99) (actual time=0.137..3.986 rows=89 loops=1)
> -> Index Scan using menu_item_restaurant_id on menu_item a (cost=0.42..177.31 rows=1 width=87) (actual time=0.130..3.769 rows=89 loops=1)
> Index Cond: (restaurant_id = 1528)
> " Filter: ((active = 'Y'::bpchar) AND (is_menu_item_available(menu_item_id, 'Y'::bpchar) = 'Y'::bpchar))"
> Rows Removed by Filter: 194
> -> Index Scan using menu_item_category_pk on menu_item_category b (cost=0.29..2.31 rows=1 width=20) (actual time=0.002..0.002 rows=1 loops=89)
> Index Cond: (menu_item_category_id = a.menu_item_category_id)
> -> Index Scan using menu_item_variant_pk on menu_item_variant c (cost=3.60..5.62 rows=1 width=45) (actual time=0.002..0.002 rows=1 loops=89)
> Index Cond: (menu_item_variant_id = (SubPlan 1))
> Filter: (a.menu_item_id = menu_item_id)
> SubPlan 1
> -> Limit (cost=3.17..3.18 rows=1 width=8) (actual time=0.009..0.009 rows=1 loops=89)
> -> Aggregate (cost=3.17..3.18 rows=1 width=8) (actual time=0.008..0.008 rows=1 loops=89)
> -> Index Scan using "idx$$_023a0001" on menu_item_variant (cost=0.43..3.15 rows=8 width=8) (actual time=0.004..0.007 rows=7 loops=89)
> Index Cond: (menu_item_id = a.menu_item_id)
> Filter: (deleted = 'N'::bpchar)
> Rows Removed by Filter: 4
> -> Index Scan using menu_item_variant_type_pk on menu_item_variant_type d (cost=0.29..0.31 rows=1 width=16) (actual time=0.002..0.002 rows=1 loops=89)
> Index Cond: (menu_item_variant_type_id = c.menu_item_variant_type_id)
> Filter: ((is_hidden)::text = 'false'::text)
> -> Index Scan using size_pk on item_size e (cost=0.29..0.31 rows=1 width=16) (actual time=0.001..0.001 rows=1 loops=89)
> Index Cond: (size_id = c.size_id)
> -> Index Scan using "restaurant_idx$$_274b003d" on restaurant f (cost=0.29..2.30 rows=1 width=12) (actual time=0.001..0.002 rows=1 loops=89)
> Index Cond: (restaurant_id = 1528)
> -> Seq Scan on mark m (cost=0.00..1.03 rows=3 width=12) (actual time=0.000..0.001 rows=3 loops=89)
> Planning Time: 1.510 ms
> Execution Time: 5.972 ms
>
>
--
--------------------------------------------------------------------
Sun Certified Enterprise Architect 1.5
Sun Certified Java Programmer 1.4
Microsoft Certified Systems Engineer 2000
http://in.linkedin.com/pub/ayub-khan/a/811/b81
mobile:+966-502674604
----------------------------------------------------------------------
It is proved that Hard Work and kowledge will get you close but attitude
will get you there. However, it's the Love
of God that will put you over the top!!
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: slow query
@ 2021-06-09 19:57 Jeff Janes <jeff.janes@gmail.com>
parent: Ayub Khan <ayub.hp@gmail.com>
0 siblings, 0 replies; 112+ messages in thread
From: Jeff Janes @ 2021-06-09 19:57 UTC (permalink / raw)
To: Ayub Khan <ayub.hp@gmail.com>; +Cc: Pgsql Performance <pgsql-performance@lists.postgresql.org>
On Tue, Jun 8, 2021 at 12:32 PM Ayub Khan <ayub.hp@gmail.com> wrote:
> In AWS RDS performance insights the client writes is high and the api
> which receives data on the mobile side is slow during load test.
>
That indicates a client or network problem.
Jeff
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: slow query
@ 2021-06-09 20:23 Ayub Khan <ayub.hp@gmail.com>
parent: Ayub Khan <ayub.hp@gmail.com>
4 siblings, 0 replies; 112+ messages in thread
From: Ayub Khan @ 2021-06-09 20:23 UTC (permalink / raw)
To: Pgsql Performance <pgsql-performance@lists.postgresql.org>
Below is the test setup
Jmeter-->(load balanced tomcat on ec2 instances)---->rds read replicas
All these are running on different ec2 instances in AWS cloud in the same
region
On Tue, 8 Jun 2021, 19:03 Ayub Khan, <ayub.hp@gmail.com> wrote:
>
> I checked all the indexes are defined on the tables however the query
> seems slow, below is the plan. Can any one give any pointers to verify ?
>
> SELECT a.menu_item_id, a.menu_item_name, a.menu_item_category_id, b.menu_item_category_desc, c.menu_item_variant_id, c.menu_item_variant_type_id, c.price, c.size_id, c.parent_menu_item_variant_id, d.menu_item_variant_type_desc, e.size_desc, f.currency_code, a.image, a.mark_id, m.mark_name
>
> FROM menu_item_category AS b, menu_item_variant AS c, menu_item_variant_type AS d, item_size AS e, restaurant AS f, menu_item AS a
>
> LEFT OUTER JOIN mark AS m ON (a.mark_id = m.mark_id) WHERE a.menu_item_category_id = b.menu_item_category_id AND a.menu_item_id = c.menu_item_id AND c.menu_item_variant_type_id = d.menu_item_variant_type_id AND d.is_hidden = 'false' AND c.size_id = e.size_id AND a.restaurant_id = f.restaurant_id AND f.restaurant_id = 1528 AND (a.menu_item_category_id = NULL OR NULL IS NULL)
>
> AND c.menu_item_variant_id = (SELECT min(menu_item_variant_id) FROM menu_item_variant WHERE menu_item_id = a.menu_item_id AND deleted = 'N' limit 1) AND a.active = 'Y'
> AND (CONCAT_WS('', ',', a.hidden_branch_ids, ',') NOT LIKE CONCAT_WS('', '%,4191,%') OR NULL IS NULL)
> AND is_menu_item_available(a.menu_item_id, 'Y') = 'Y'
>
> ORDER BY a.row_order, menu_item_id;
>
>
> below is the plan
>
>
> Sort (cost=189.27..189.27 rows=1 width=152) (actual time=5.876..5.885 rows=89 loops=1)
> " Sort Key: a.row_order, a.menu_item_id"
> Sort Method: quicksort Memory: 48kB
> -> Nested Loop Left Join (cost=5.19..189.26 rows=1 width=152) (actual time=0.188..5.809 rows=89 loops=1)
> Join Filter: (a.mark_id = m.mark_id)
> Rows Removed by Join Filter: 267
> -> Nested Loop (cost=5.19..188.19 rows=1 width=148) (actual time=0.181..5.629 rows=89 loops=1)
> -> Nested Loop (cost=4.90..185.88 rows=1 width=152) (actual time=0.174..5.443 rows=89 loops=1)
> -> Nested Loop (cost=4.61..185.57 rows=1 width=144) (actual time=0.168..5.272 rows=89 loops=1)
> -> Nested Loop (cost=4.32..185.25 rows=1 width=136) (actual time=0.162..5.066 rows=89 loops=1)
> -> Nested Loop (cost=0.71..179.62 rows=1 width=99) (actual time=0.137..3.986 rows=89 loops=1)
> -> Index Scan using menu_item_restaurant_id on menu_item a (cost=0.42..177.31 rows=1 width=87) (actual time=0.130..3.769 rows=89 loops=1)
> Index Cond: (restaurant_id = 1528)
> " Filter: ((active = 'Y'::bpchar) AND (is_menu_item_available(menu_item_id, 'Y'::bpchar) = 'Y'::bpchar))"
> Rows Removed by Filter: 194
> -> Index Scan using menu_item_category_pk on menu_item_category b (cost=0.29..2.31 rows=1 width=20) (actual time=0.002..0.002 rows=1 loops=89)
> Index Cond: (menu_item_category_id = a.menu_item_category_id)
> -> Index Scan using menu_item_variant_pk on menu_item_variant c (cost=3.60..5.62 rows=1 width=45) (actual time=0.002..0.002 rows=1 loops=89)
> Index Cond: (menu_item_variant_id = (SubPlan 1))
> Filter: (a.menu_item_id = menu_item_id)
> SubPlan 1
> -> Limit (cost=3.17..3.18 rows=1 width=8) (actual time=0.009..0.009 rows=1 loops=89)
> -> Aggregate (cost=3.17..3.18 rows=1 width=8) (actual time=0.008..0.008 rows=1 loops=89)
> -> Index Scan using "idx$$_023a0001" on menu_item_variant (cost=0.43..3.15 rows=8 width=8) (actual time=0.004..0.007 rows=7 loops=89)
> Index Cond: (menu_item_id = a.menu_item_id)
> Filter: (deleted = 'N'::bpchar)
> Rows Removed by Filter: 4
> -> Index Scan using menu_item_variant_type_pk on menu_item_variant_type d (cost=0.29..0.31 rows=1 width=16) (actual time=0.002..0.002 rows=1 loops=89)
> Index Cond: (menu_item_variant_type_id = c.menu_item_variant_type_id)
> Filter: ((is_hidden)::text = 'false'::text)
> -> Index Scan using size_pk on item_size e (cost=0.29..0.31 rows=1 width=16) (actual time=0.001..0.001 rows=1 loops=89)
> Index Cond: (size_id = c.size_id)
> -> Index Scan using "restaurant_idx$$_274b003d" on restaurant f (cost=0.29..2.30 rows=1 width=12) (actual time=0.001..0.002 rows=1 loops=89)
> Index Cond: (restaurant_id = 1528)
> -> Seq Scan on mark m (cost=0.00..1.03 rows=3 width=12) (actual time=0.000..0.001 rows=3 loops=89)
> Planning Time: 1.510 ms
> Execution Time: 5.972 ms
>
>
^ permalink raw reply [nested|flat] 112+ messages in thread
end of thread, other threads:[~2021-06-09 20:23 UTC | newest]
Thread overview: 112+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2003-02-21 16:30 slow query Oleg Lebedev <oleg.lebedev@waterford.org>
2003-02-22 02:20 ` Tom Lane <tgl@sss.pgh.pa.us>
2003-02-23 20:52 ` Josh Berkus <josh@agliodbs.com>
2003-02-23 21:05 ` Ryan Bradetich <rbradetich@uswest.net>
2003-02-23 21:47 ` Josh Berkus <josh@agliodbs.com>
2003-02-24 02:28 ` Tom Lane <tgl@sss.pgh.pa.us>
2003-02-24 15:59 Re: slow query Oleg Lebedev <oleg.lebedev@waterford.org>
2003-02-24 16:58 ` Robert Treat <xzilla@users.sourceforge.net>
2003-02-24 17:27 ` Clarence Gardner <clarence@silcom.com>
2003-02-24 18:30 ` Robert Treat <xzilla@users.sourceforge.net>
2003-02-24 18:45 ` Josh Berkus <josh@agliodbs.com>
2003-02-24 19:58 ` Robert Treat <xzilla@users.sourceforge.net>
2003-02-24 20:50 ` Tom Lane <tgl@sss.pgh.pa.us>
2003-02-25 15:28 ` Robert Treat <xzilla@users.sourceforge.net>
2003-02-25 16:07 ` Tom Lane <tgl@sss.pgh.pa.us>
2003-02-25 16:36 ` Peter Childs <blue.dragon@blueyonder.co.uk>
2003-02-24 18:39 ` Rod Taylor <rbt@rbt.ca>
2003-02-24 18:51 ` Andrew Sullivan <andrew@libertyrms.info>
2003-02-24 17:07 Re: slow query Oleg Lebedev <oleg.lebedev@waterford.org>
2003-02-24 17:44 Re: slow query Oleg Lebedev <oleg.lebedev@waterford.org>
2003-03-24 17:48 Slow query Oleg Lebedev <oleg.lebedev@waterford.org>
2003-03-24 18:54 ` Re: Slow query Josh Berkus <josh@agliodbs.com>
2003-03-24 19:03 ` Re: Slow query Stephan Szabo <sszabo@megazone23.bigpanda.com>
2003-03-24 19:47 ` Re: Slow query Josh Berkus <josh@agliodbs.com>
2003-03-24 20:48 ` Re: Slow query Tom Lane <tgl@sss.pgh.pa.us>
2003-03-24 19:02 Re: Slow query Oleg Lebedev <oleg.lebedev@waterford.org>
2003-03-24 19:20 Re: Slow query Oleg Lebedev <oleg.lebedev@waterford.org>
2003-03-24 20:28 Re: Slow query Oleg Lebedev <oleg.lebedev@waterford.org>
2003-03-24 21:46 Re: Slow query Oleg Lebedev <oleg.lebedev@waterford.org>
2003-03-24 22:08 ` Re: Slow query Tom Lane <tgl@sss.pgh.pa.us>
2003-03-24 22:37 Re: Slow query Oleg Lebedev <oleg.lebedev@waterford.org>
2003-03-25 03:26 Re: Slow query Nikolaus Dilger <nikolaus@dilger.cc>
2003-09-09 06:27 Slow query? Waruna Geekiyanage <waruna@nirmani.com>
2004-02-24 13:17 Slow query Ivan Voras <ivoras@fer.hr>
2004-03-01 16:26 ` Re: Slow query scott.marlowe <scott.marlowe@ihs.com>
2004-10-22 19:06 Slow query Roger Ging <rging@paccomsys.com>
2004-10-22 20:37 ` Re: Slow query Tom Lane <tgl@sss.pgh.pa.us>
2004-10-23 12:08 ` Re: Slow query Joshua Marsh <icub3d@gmail.com>
2005-07-14 00:06 Slow Query Marc McIntyre <mmcintyre@squiz.net>
2005-07-14 11:22 ` Re: Slow Query Ragnar Hafstað <gnari@simnet.is>
2006-11-21 19:52 ` Re: Slow Query Joe Lester <joe_lester@sweetwater.com>
2006-02-22 19:16 Slow query Jeremy Haile <jhaile@fastmail.fm>
2006-02-23 17:25 ` Re: Slow query Tom Lane <tgl@sss.pgh.pa.us>
2006-11-21 17:03 Slow Query Joe Lester <joe_lester@sweetwater.com>
2007-07-02 10:07 slow query Vidhya Bondre <meetvbondre@gmail.com>
2007-07-02 13:20 ` Nis Jørgensen <nis@superlativ.dk>
2007-09-01 17:29 Slow Query Shawn <postgres@xmtservices.net>
2007-09-01 18:09 ` Re: Slow Query Tom Lane <tgl@sss.pgh.pa.us>
2007-09-01 20:18 ` Re: Slow Query Shawn <postgres@xmtservices.net>
2007-09-02 00:35 ` Re: Slow Query Shawn <postgres@xmtservices.net>
2007-09-02 03:00 ` Re: Slow Query Tom Lane <tgl@sss.pgh.pa.us>
2007-09-03 16:57 ` Re: Slow Query Shawn <postgres@xmtservices.net>
2007-09-03 18:32 ` Re: Slow Query Kevin Grittner <Kevin.Grittner@wicourts.gov>
2007-09-02 15:49 ` Re: Slow Query Kevin Grittner <Kevin.Grittner@wicourts.gov>
2007-09-03 16:15 ` Re: Slow Query Shawn <postgres@xmtservices.net>
2007-09-03 18:07 ` Re: Slow Query Kevin Grittner <Kevin.Grittner@wicourts.gov>
2007-09-03 23:53 ` Re: Slow Query Shawn <postgres@xmtservices.net>
2007-09-04 00:49 ` Re: Slow Query Tom Lane <tgl@sss.pgh.pa.us>
2007-09-10 01:31 ` Re: Slow Query Shawn <postgres@xmtservices.net>
2007-09-04 02:40 ` Re: Slow Query Kevin Grittner <Kevin.Grittner@wicourts.gov>
2007-12-11 15:02 Slow Query Pallav Kalva <pkalva@livedatagroup.com>
2010-06-04 01:45 slow query Anj Adu <fotographs@gmail.com>
2010-06-04 08:13 ` tv@fuzzy.cz
2010-06-04 17:41 ` Anj Adu <fotographs@gmail.com>
2010-06-04 18:01 ` Anj Adu <fotographs@gmail.com>
2010-06-04 18:21 ` Anj Adu <fotographs@gmail.com>
2010-06-05 09:38 ` Scott Marlowe <scott.marlowe@gmail.com>
2010-06-05 14:02 ` Anj Adu <fotographs@gmail.com>
2010-06-05 21:25 ` Scott Marlowe <scott.marlowe@gmail.com>
2010-06-04 09:00 ` Matthew Wakeling <matthew@flymine.org>
2010-06-04 17:25 ` Anj Adu <fotographs@gmail.com>
2010-06-05 09:02 ` hubert depesz lubaczewski <depesz@depesz.com>
2010-08-27 00:03 Slow Query Ozer, Pam <pozer@automotive.com>
2010-08-27 00:18 ` Re: Slow Query Scott Marlowe <scott.marlowe@gmail.com>
2010-08-27 00:19 ` Re: Slow Query Bob Lunney <bob_lunney@yahoo.com>
2010-08-27 14:17 ` Re: Slow Query Kevin Grittner <Kevin.Grittner@wicourts.gov>
2010-08-27 17:30 ` Re: Slow Query Greg Smith <greg@2ndquadrant.com>
2010-08-27 17:44 ` Re: Slow Query Ozer, Pam <pozer@automotive.com>
2014-03-07 06:05 Slow query Bikram Kesari Naik <bikram.naik@onmobile.com>
2014-03-07 06:23 ` Re: Slow query David Johnston <polobo@yahoo.com>
2014-03-07 06:43 ` Re: Slow query Bikram Kesari Naik <bikram.naik@onmobile.com>
2014-03-07 07:18 ` Re: Slow query David Johnston <polobo@yahoo.com>
2014-09-23 12:21 Slow query Ross Elliott <ross@wearthefoxhat.com>
2014-09-23 13:05 ` Re: Slow query David G Johnston <david.g.johnston@gmail.com>
2015-02-11 06:41 slow query Sathish Nelson <satson.csc17@gmail.com>
2015-02-11 13:33 ` Torsten Förtsch <torsten.foertsch@gmx.net>
2015-02-11 13:44 ` Tom Lane <tgl@sss.pgh.pa.us>
2015-08-12 02:34 Slow Query robbyc <robcampbell73@gmail.com>
2015-08-12 04:08 ` Re: Slow Query Venkata Balaji N <nag1010@gmail.com>
2015-08-12 05:29 ` Re: Slow Query robbyc <robcampbell73@gmail.com>
2015-08-12 06:08 ` Re: Slow Query Venkata Balaji N <nag1010@gmail.com>
2015-08-12 11:34 ` Re: Slow Query Vik Fearing <vik@2ndquadrant.fr>
2015-08-13 00:50 ` Re: Slow Query robbyc <robcampbell73@gmail.com>
2015-08-13 09:21 ` Re: Slow Query 林士博 <lin@repica.co.jp>
2015-08-14 03:15 ` Re: Slow Query robbyc <robcampbell73@gmail.com>
2015-08-14 05:10 ` Re: Slow Query 林士博 <lin@repica.co.jp>
2015-08-14 06:09 ` Re: Slow Query 林士博 <lin@repica.co.jp>
2020-04-03 16:03 slow query dangal <danielito.gallo@gmail.com>
2020-04-03 16:07 ` Justin Pryzby <pryzby@telsasoft.com>
2020-04-03 16:18 ` dangal <danielito.gallo@gmail.com>
2020-04-07 10:20 ` Michael Christofides <michristofides@gmail.com>
2020-10-14 17:30 Slow Query Parth Shah <parth@polimorphic.com>
2020-10-14 19:17 ` Re: Slow Query Michael Lewis <mlewis@entrata.com>
2020-10-15 01:11 ` Re: Slow Query Parth Shah <parth@polimorphic.com>
2020-10-15 04:51 ` Re: Slow Query Michael Lewis <mlewis@entrata.com>
2021-06-08 16:03 slow query Ayub Khan <ayub.hp@gmail.com>
2021-06-08 16:08 ` Christophe Pettus <xof@thebuild.com>
2021-06-08 16:32 ` Ayub Khan <ayub.hp@gmail.com>
2021-06-09 19:57 ` Jeff Janes <jeff.janes@gmail.com>
2021-06-08 17:14 ` Tom Lane <tgl@sss.pgh.pa.us>
2021-06-08 17:43 ` Ayub Khan <ayub.hp@gmail.com>
2021-06-09 20:23 ` Ayub Khan <ayub.hp@gmail.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox