pg.ddx.io pgsql-performance@postgresql.org mailing list archive
help / color / mirror / Atom feedPerformance issues
59+ messages / 18 participants
[nested] [flat]
* Performance issues
@ 2011-03-07 13:45 Andreas Forø Tollefsen <andreasft@gmail.com>
2011-03-07 14:01 ` Re: Performance issues Kenneth Marshall <ktm@rice.edu>
0 siblings, 1 reply; 59+ messages in thread
From: Andreas Forø Tollefsen @ 2011-03-07 13:45 UTC (permalink / raw)
To: pgsql-performance
Hi,
I am running Postgresql 8.4.7 with Postgis 2.0 (for raster support).
Server is mainly 1 user for spatial data processing. This involves queries
that can take hours.
This is running on a ubuntu 10.10 Server with Core2Duo 6600 @ 2.4 GHZ, 6 GB
RAM.
My postgresql.conf:
# - Memory -
shared_buffers = 1024MB # min 128kB
# (change requires restart)
temp_buffers = 256MB # min 800kB
#max_prepared_transactions = 0 # zero disables the feature
# (change requires restart)
# Note: Increasing max_prepared_transactions costs ~600 bytes of shared
memory
# per transaction slot, plus lock space (see max_locks_per_transaction).
# It is not advisable to set max_prepared_transactions nonzero unless you
# actively intend to use prepared transactions.
work_mem = 1024MB # min 64kB
maintenance_work_mem = 256MB # min 1MB
max_stack_depth = 7MB # min 100kB
wal_buffers = 8MB
effective_cache_size = 3072MB
Everything else is default.
My Pgbench results:
/usr/lib/postgresql/8.4/bin/pgbench -T 60 test1
starting vacuum...end.
transaction type: TPC-B (sort of)
scaling factor: 1
query mode: simple
number of clients: 1
duration: 60 s
number of transactions actually processed: 7004
tps = 116.728199 (including connections establishing)
tps = 116.733012 (excluding connections establishing)
My question is if these are acceptable results, or if someone can recommend
settings which will improve my servers performance.
Andreas
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Performance issues
2011-03-07 13:45 Performance issues Andreas Forø Tollefsen <andreasft@gmail.com>
@ 2011-03-07 14:01 ` Kenneth Marshall <ktm@rice.edu>
2011-03-07 14:17 ` Re: Performance issues Andreas Forø Tollefsen <andreasft@gmail.com>
0 siblings, 1 reply; 59+ messages in thread
From: Kenneth Marshall @ 2011-03-07 14:01 UTC (permalink / raw)
To: Andreas For? Tollefsen <andreasft@gmail.com>; +Cc: pgsql-performance
On Mon, Mar 07, 2011 at 02:45:03PM +0100, Andreas For? Tollefsen wrote:
> Hi,
>
> I am running Postgresql 8.4.7 with Postgis 2.0 (for raster support).
> Server is mainly 1 user for spatial data processing. This involves queries
> that can take hours.
>
> This is running on a ubuntu 10.10 Server with Core2Duo 6600 @ 2.4 GHZ, 6 GB
> RAM.
>
> My postgresql.conf:
> # - Memory -
> shared_buffers = 1024MB # min 128kB
> # (change requires restart)
> temp_buffers = 256MB # min 800kB
> #max_prepared_transactions = 0 # zero disables the feature
> # (change requires restart)
> # Note: Increasing max_prepared_transactions costs ~600 bytes of shared
> memory
> # per transaction slot, plus lock space (see max_locks_per_transaction).
> # It is not advisable to set max_prepared_transactions nonzero unless you
> # actively intend to use prepared transactions.
> work_mem = 1024MB # min 64kB
> maintenance_work_mem = 256MB # min 1MB
> max_stack_depth = 7MB # min 100kB
> wal_buffers = 8MB
> effective_cache_size = 3072MB
>
> Everything else is default.
>
> My Pgbench results:
> /usr/lib/postgresql/8.4/bin/pgbench -T 60 test1
> starting vacuum...end.
> transaction type: TPC-B (sort of)
> scaling factor: 1
> query mode: simple
> number of clients: 1
> duration: 60 s
> number of transactions actually processed: 7004
> tps = 116.728199 (including connections establishing)
> tps = 116.733012 (excluding connections establishing)
>
>
> My question is if these are acceptable results, or if someone can recommend
> settings which will improve my servers performance.
>
> Andreas
Your results are I/O limited. Depending upon your requirements,
you may be able to turn off synchronous_commit which can help.
Your actual workload may be able to use batching to help as well.
Your work_mem looks pretty darn high for a 6GB system.
Cheers,
Ken
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Performance issues
2011-03-07 13:45 Performance issues Andreas Forø Tollefsen <andreasft@gmail.com>
2011-03-07 14:01 ` Re: Performance issues Kenneth Marshall <ktm@rice.edu>
@ 2011-03-07 14:17 ` Andreas Forø Tollefsen <andreasft@gmail.com>
2011-03-07 14:22 ` Re: Performance issues Kenneth Marshall <ktm@rice.edu>
0 siblings, 1 reply; 59+ messages in thread
From: Andreas Forø Tollefsen @ 2011-03-07 14:17 UTC (permalink / raw)
To: Kenneth Marshall <ktm@rice.edu>; +Cc: pgsql-performance
Thanks, Ken.
It seems like the tip to turn off synchronous_commit did the trick:
/usr/lib/postgresql/8.4/bin/pgbench -T 60 test1
starting vacuum...end.
transaction type: TPC-B (sort of)
scaling factor: 1
query mode: simple
number of clients: 1
duration: 60 s
number of transactions actually processed: 86048
tps = 1434.123199 (including connections establishing)
tps = 1434.183362 (excluding connections establishing)
Is this acceptable compared to others when considering my setup?
Cheers,
Andreas
2011/3/7 Kenneth Marshall <ktm@rice.edu>
> On Mon, Mar 07, 2011 at 02:45:03PM +0100, Andreas For? Tollefsen wrote:
> > Hi,
> >
> > I am running Postgresql 8.4.7 with Postgis 2.0 (for raster support).
> > Server is mainly 1 user for spatial data processing. This involves
> queries
> > that can take hours.
> >
> > This is running on a ubuntu 10.10 Server with Core2Duo 6600 @ 2.4 GHZ, 6
> GB
> > RAM.
> >
> > My postgresql.conf:
> > # - Memory -
> > shared_buffers = 1024MB # min 128kB
> > # (change requires restart)
> > temp_buffers = 256MB # min 800kB
> > #max_prepared_transactions = 0 # zero disables the feature
> > # (change requires restart)
> > # Note: Increasing max_prepared_transactions costs ~600 bytes of shared
> > memory
> > # per transaction slot, plus lock space (see max_locks_per_transaction).
> > # It is not advisable to set max_prepared_transactions nonzero unless you
> > # actively intend to use prepared transactions.
> > work_mem = 1024MB # min 64kB
> > maintenance_work_mem = 256MB # min 1MB
> > max_stack_depth = 7MB # min 100kB
> > wal_buffers = 8MB
> > effective_cache_size = 3072MB
> >
> > Everything else is default.
> >
> > My Pgbench results:
> > /usr/lib/postgresql/8.4/bin/pgbench -T 60 test1
> > starting vacuum...end.
> > transaction type: TPC-B (sort of)
> > scaling factor: 1
> > query mode: simple
> > number of clients: 1
> > duration: 60 s
> > number of transactions actually processed: 7004
> > tps = 116.728199 (including connections establishing)
> > tps = 116.733012 (excluding connections establishing)
> >
> >
> > My question is if these are acceptable results, or if someone can
> recommend
> > settings which will improve my servers performance.
> >
> > Andreas
>
> Your results are I/O limited. Depending upon your requirements,
> you may be able to turn off synchronous_commit which can help.
> Your actual workload may be able to use batching to help as well.
> Your work_mem looks pretty darn high for a 6GB system.
>
> Cheers,
> Ken
>
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Performance issues
2011-03-07 13:45 Performance issues Andreas Forø Tollefsen <andreasft@gmail.com>
2011-03-07 14:01 ` Re: Performance issues Kenneth Marshall <ktm@rice.edu>
2011-03-07 14:17 ` Re: Performance issues Andreas Forø Tollefsen <andreasft@gmail.com>
@ 2011-03-07 14:22 ` Kenneth Marshall <ktm@rice.edu>
2011-03-07 14:29 ` Re: Performance issues Andreas Forø Tollefsen <andreasft@gmail.com>
0 siblings, 1 reply; 59+ messages in thread
From: Kenneth Marshall @ 2011-03-07 14:22 UTC (permalink / raw)
To: Andreas For? Tollefsen <andreasft@gmail.com>; +Cc: pgsql-performance
On Mon, Mar 07, 2011 at 03:17:05PM +0100, Andreas For? Tollefsen wrote:
> Thanks, Ken.
>
> It seems like the tip to turn off synchronous_commit did the trick:
>
> /usr/lib/postgresql/8.4/bin/pgbench -T 60 test1
> starting vacuum...end.
> transaction type: TPC-B (sort of)
> scaling factor: 1
> query mode: simple
> number of clients: 1
> duration: 60 s
> number of transactions actually processed: 86048
> tps = 1434.123199 (including connections establishing)
> tps = 1434.183362 (excluding connections establishing)
>
> Is this acceptable compared to others when considering my setup?
>
> Cheers,
> Andreas
>
These are typical results for synchronous_commit off. The caveat
is you must be able to handle loosing transactions if you have a
database crash, but your database is still intact. This differs
from turning fsync off in which a crash means you would need to
restore from a backup.
Cheers,
Ken
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Performance issues
2011-03-07 13:45 Performance issues Andreas Forø Tollefsen <andreasft@gmail.com>
2011-03-07 14:01 ` Re: Performance issues Kenneth Marshall <ktm@rice.edu>
2011-03-07 14:17 ` Re: Performance issues Andreas Forø Tollefsen <andreasft@gmail.com>
2011-03-07 14:22 ` Re: Performance issues Kenneth Marshall <ktm@rice.edu>
@ 2011-03-07 14:29 ` Andreas Forø Tollefsen <andreasft@gmail.com>
2011-03-07 15:34 ` Re: Performance issues Oleg Bartunov <oleg@sai.msu.su>
0 siblings, 1 reply; 59+ messages in thread
From: Andreas Forø Tollefsen @ 2011-03-07 14:29 UTC (permalink / raw)
To: Kenneth Marshall <ktm@rice.edu>; +Cc: pgsql-performance
Ok. Cheers. I will do some more testing on my heavy PostGIS queries which
often takes hours to complete.
Thanks.
Andreas
2011/3/7 Kenneth Marshall <ktm@rice.edu>
> On Mon, Mar 07, 2011 at 03:17:05PM +0100, Andreas For? Tollefsen wrote:
> > Thanks, Ken.
> >
> > It seems like the tip to turn off synchronous_commit did the trick:
> >
> > /usr/lib/postgresql/8.4/bin/pgbench -T 60 test1
> > starting vacuum...end.
> > transaction type: TPC-B (sort of)
> > scaling factor: 1
> > query mode: simple
> > number of clients: 1
> > duration: 60 s
> > number of transactions actually processed: 86048
> > tps = 1434.123199 (including connections establishing)
> > tps = 1434.183362 (excluding connections establishing)
> >
> > Is this acceptable compared to others when considering my setup?
> >
> > Cheers,
> > Andreas
> >
>
>
> These are typical results for synchronous_commit off. The caveat
> is you must be able to handle loosing transactions if you have a
> database crash, but your database is still intact. This differs
> from turning fsync off in which a crash means you would need to
> restore from a backup.
>
> Cheers,
> Ken
>
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Performance issues
2011-03-07 13:45 Performance issues Andreas Forø Tollefsen <andreasft@gmail.com>
2011-03-07 14:01 ` Re: Performance issues Kenneth Marshall <ktm@rice.edu>
2011-03-07 14:17 ` Re: Performance issues Andreas Forø Tollefsen <andreasft@gmail.com>
2011-03-07 14:22 ` Re: Performance issues Kenneth Marshall <ktm@rice.edu>
2011-03-07 14:29 ` Re: Performance issues Andreas Forø Tollefsen <andreasft@gmail.com>
@ 2011-03-07 15:34 ` Oleg Bartunov <oleg@sai.msu.su>
2011-03-07 21:49 ` Re: Performance issues Andreas Forø Tollefsen <andreasft@gmail.com>
0 siblings, 1 reply; 59+ messages in thread
From: Oleg Bartunov @ 2011-03-07 15:34 UTC (permalink / raw)
To: Andreas Forø Tollefsen <andreasft@gmail.com>; +Cc: Kenneth Marshall <ktm@rice.edu>; pgsql-performance
On Mon, 7 Mar 2011, Andreas For? Tollefsen wrote:
> Ok. Cheers. I will do some more testing on my heavy PostGIS queries which
> often takes hours to complete.
I'd like to see hours long queries :) EXPLAIN ANALYZE
>
> Thanks.
> Andreas
>
> 2011/3/7 Kenneth Marshall <ktm@rice.edu>
>
>> On Mon, Mar 07, 2011 at 03:17:05PM +0100, Andreas For? Tollefsen wrote:
>>> Thanks, Ken.
>>>
>>> It seems like the tip to turn off synchronous_commit did the trick:
>>>
>>> /usr/lib/postgresql/8.4/bin/pgbench -T 60 test1
>>> starting vacuum...end.
>>> transaction type: TPC-B (sort of)
>>> scaling factor: 1
>>> query mode: simple
>>> number of clients: 1
>>> duration: 60 s
>>> number of transactions actually processed: 86048
>>> tps = 1434.123199 (including connections establishing)
>>> tps = 1434.183362 (excluding connections establishing)
>>>
>>> Is this acceptable compared to others when considering my setup?
>>>
>>> Cheers,
>>> Andreas
>>>
>>
>>
>> These are typical results for synchronous_commit off. The caveat
>> is you must be able to handle loosing transactions if you have a
>> database crash, but your database is still intact. This differs
>> from turning fsync off in which a crash means you would need to
>> restore from a backup.
>>
>> Cheers,
>> Ken
>>
>
Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Performance issues
2011-03-07 13:45 Performance issues Andreas Forø Tollefsen <andreasft@gmail.com>
2011-03-07 14:01 ` Re: Performance issues Kenneth Marshall <ktm@rice.edu>
2011-03-07 14:17 ` Re: Performance issues Andreas Forø Tollefsen <andreasft@gmail.com>
2011-03-07 14:22 ` Re: Performance issues Kenneth Marshall <ktm@rice.edu>
2011-03-07 14:29 ` Re: Performance issues Andreas Forø Tollefsen <andreasft@gmail.com>
2011-03-07 15:34 ` Re: Performance issues Oleg Bartunov <oleg@sai.msu.su>
@ 2011-03-07 21:49 ` Andreas Forø Tollefsen <andreasft@gmail.com>
2011-03-07 23:29 ` Re: Performance issues David Kerr <dmk@mr-paradox.net>
2011-03-08 00:38 ` Re: Performance issues Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 2 replies; 59+ messages in thread
From: Andreas Forø Tollefsen @ 2011-03-07 21:49 UTC (permalink / raw)
To: Oleg Bartunov <oleg@sai.msu.su>; +Cc: Kenneth Marshall <ktm@rice.edu>; pgsql-performance
The synchronous_commit off increased the TPS, but not the speed of the below
query.
Oleg:
This is a query i am working on now. It creates an intersection of two
geometries. One is a grid of 0.5 x 0.5 decimal degree sized cells, while the
other is the country geometries of all countries in the world for a certain
year.
priogrid=# EXPLAIN ANALYZE SELECT priogrid_land.gid, gwcode,
ST_Intersection(pri
ogrid_land.cell, cshapeswdate.geom) FROM priogrid_land, cshapeswdate WHERE
ST_In
tersects(priogrid_land.cell, cshapeswdate.geom);
QUERY
PLAN
--------------------------------------------------------------------------------
------------------------------------------------------------------
Nested Loop (cost=0.00..12644.85 rows=43351 width=87704) (actual
time=1.815..7
074973.711 rows=130331 loops=1)
Join Filter: _st_intersects(priogrid_land.cell, cshapeswdate.geom)
-> Seq Scan on cshapeswdate (cost=0.00..14.42 rows=242 width=87248)
(actual
time=0.007..0.570 rows=242 loops=1)
-> Index Scan using idx_priogrid_land_cell on priogrid_land
(cost=0.00..7.1
5 rows=1 width=456) (actual time=0.069..5.604 rows=978 loops=242)
Index Cond: (priogrid_land.cell && cshapeswdate.geom)
Total runtime: 7075188.549 ms
(6 rows)
2011/3/7 Oleg Bartunov <oleg@sai.msu.su>
> On Mon, 7 Mar 2011, Andreas For? Tollefsen wrote:
>
> Ok. Cheers. I will do some more testing on my heavy PostGIS queries which
>> often takes hours to complete.
>>
>
> I'd like to see hours long queries :) EXPLAIN ANALYZE
>
>
>
>> Thanks.
>> Andreas
>>
>> 2011/3/7 Kenneth Marshall <ktm@rice.edu>
>>
>> On Mon, Mar 07, 2011 at 03:17:05PM +0100, Andreas For? Tollefsen wrote:
>>>
>>>> Thanks, Ken.
>>>>
>>>> It seems like the tip to turn off synchronous_commit did the trick:
>>>>
>>>> /usr/lib/postgresql/8.4/bin/pgbench -T 60 test1
>>>> starting vacuum...end.
>>>> transaction type: TPC-B (sort of)
>>>> scaling factor: 1
>>>> query mode: simple
>>>> number of clients: 1
>>>> duration: 60 s
>>>> number of transactions actually processed: 86048
>>>> tps = 1434.123199 (including connections establishing)
>>>> tps = 1434.183362 (excluding connections establishing)
>>>>
>>>> Is this acceptable compared to others when considering my setup?
>>>>
>>>> Cheers,
>>>> Andreas
>>>>
>>>>
>>>
>>> These are typical results for synchronous_commit off. The caveat
>>> is you must be able to handle loosing transactions if you have a
>>> database crash, but your database is still intact. This differs
>>> from turning fsync off in which a crash means you would need to
>>> restore from a backup.
>>>
>>> Cheers,
>>> Ken
>>>
>>>
>>
> Regards,
> Oleg
> _____________________________________________________________
> Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
> Sternberg Astronomical Institute, Moscow University, Russia
> Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
> phone: +007(495)939-16-83, +007(495)939-23-83
>
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Performance issues
2011-03-07 13:45 Performance issues Andreas Forø Tollefsen <andreasft@gmail.com>
2011-03-07 14:01 ` Re: Performance issues Kenneth Marshall <ktm@rice.edu>
2011-03-07 14:17 ` Re: Performance issues Andreas Forø Tollefsen <andreasft@gmail.com>
2011-03-07 14:22 ` Re: Performance issues Kenneth Marshall <ktm@rice.edu>
2011-03-07 14:29 ` Re: Performance issues Andreas Forø Tollefsen <andreasft@gmail.com>
2011-03-07 15:34 ` Re: Performance issues Oleg Bartunov <oleg@sai.msu.su>
2011-03-07 21:49 ` Re: Performance issues Andreas Forø Tollefsen <andreasft@gmail.com>
@ 2011-03-07 23:29 ` David Kerr <dmk@mr-paradox.net>
1 sibling, 0 replies; 59+ messages in thread
From: David Kerr @ 2011-03-07 23:29 UTC (permalink / raw)
To: Andreas Forø Tollefsen <andreasft@gmail.com>; +Cc: Oleg Bartunov <oleg@sai.msu.su>; Kenneth Marshall <ktm@rice.edu>; pgsql-performance
On Mon, Mar 07, 2011 at 10:49:48PM +0100, Andreas For Tollefsen wrote:
- The synchronous_commit off increased the TPS, but not the speed of the below
- query.
-
- Oleg:
- This is a query i am working on now. It creates an intersection of two
- geometries. One is a grid of 0.5 x 0.5 decimal degree sized cells, while the
- other is the country geometries of all countries in the world for a certain
- year.
-
- priogrid=# EXPLAIN ANALYZE SELECT priogrid_land.gid, gwcode,
- ST_Intersection(pri
- ogrid_land.cell, cshapeswdate.geom) FROM priogrid_land, cshapeswdate WHERE
- ST_In
- tersects(priogrid_land.cell, cshapeswdate.geom);
- QUERY
- PLAN
-
- --------------------------------------------------------------------------------
- ------------------------------------------------------------------
- Nested Loop (cost=0.00..12644.85 rows=43351 width=87704) (actual
- time=1.815..7
- 074973.711 rows=130331 loops=1)
- Join Filter: _st_intersects(priogrid_land.cell, cshapeswdate.geom)
- -> Seq Scan on cshapeswdate (cost=0.00..14.42 rows=242 width=87248)
- (actual
- time=0.007..0.570 rows=242 loops=1)
- -> Index Scan using idx_priogrid_land_cell on priogrid_land
- (cost=0.00..7.1
- 5 rows=1 width=456) (actual time=0.069..5.604 rows=978 loops=242)
- Index Cond: (priogrid_land.cell && cshapeswdate.geom)
- Total runtime: 7075188.549 ms
- (6 rows)
Your estimated and actuals are way off, have you analyzed those tables?
Dave
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Performance issues
2011-03-07 13:45 Performance issues Andreas Forø Tollefsen <andreasft@gmail.com>
2011-03-07 14:01 ` Re: Performance issues Kenneth Marshall <ktm@rice.edu>
2011-03-07 14:17 ` Re: Performance issues Andreas Forø Tollefsen <andreasft@gmail.com>
2011-03-07 14:22 ` Re: Performance issues Kenneth Marshall <ktm@rice.edu>
2011-03-07 14:29 ` Re: Performance issues Andreas Forø Tollefsen <andreasft@gmail.com>
2011-03-07 15:34 ` Re: Performance issues Oleg Bartunov <oleg@sai.msu.su>
2011-03-07 21:49 ` Re: Performance issues Andreas Forø Tollefsen <andreasft@gmail.com>
@ 2011-03-08 00:38 ` Tom Lane <tgl@sss.pgh.pa.us>
2011-03-08 08:42 ` Re: Performance issues Andreas Forø Tollefsen <andreasft@gmail.com>
1 sibling, 1 reply; 59+ messages in thread
From: Tom Lane @ 2011-03-08 00:38 UTC (permalink / raw)
To: Andreas Forø Tollefsen <andreasft@gmail.com>; +Cc: Oleg Bartunov <oleg@sai.msu.su>; Kenneth Marshall <ktm@rice.edu>; pgsql-performance
=?ISO-8859-1?Q?Andreas_For=F8_Tollefsen?= <andreasft@gmail.com> writes:
> This is a query i am working on now. It creates an intersection of two
> geometries. One is a grid of 0.5 x 0.5 decimal degree sized cells, while the
> other is the country geometries of all countries in the world for a certain
> year.
Hm, are you sure your data is right? Because the actual rowcounts imply
that each country intersects about half of the grid cells, which doesn't
seem right.
> priogrid=# EXPLAIN ANALYZE SELECT priogrid_land.gid, gwcode,
> ST_Intersection(pri
> ogrid_land.cell, cshapeswdate.geom) FROM priogrid_land, cshapeswdate WHERE
> ST_Intersects(priogrid_land.cell, cshapeswdate.geom);
> QUERY
> PLAN
> --------------------------------------------------------------------------------
> ------------------------------------------------------------------
> Nested Loop (cost=0.00..12644.85 rows=43351 width=87704) (actual
> time=1.815..7
> 074973.711 rows=130331 loops=1)
> Join Filter: _st_intersects(priogrid_land.cell, cshapeswdate.geom)
> -> Seq Scan on cshapeswdate (cost=0.00..14.42 rows=242 width=87248)
> (actual
> time=0.007..0.570 rows=242 loops=1)
> -> Index Scan using idx_priogrid_land_cell on priogrid_land
> (cost=0.00..7.1
> 5 rows=1 width=456) (actual time=0.069..5.604 rows=978 loops=242)
> Index Cond: (priogrid_land.cell && cshapeswdate.geom)
> Total runtime: 7075188.549 ms
> (6 rows)
AFAICT, all of the runtime is going into calculating the ST_Intersects
and/or ST_Intersection functions. The two scans are only accounting for
perhaps 5.5 seconds, and the join infrastructure isn't going to be
terribly expensive, so it's got to be those functions. Not knowing much
about PostGIS, I don't know if the functions themselves can be expected
to be really slow. If it's not them, it could be the cost of fetching
their arguments --- in particular, I bet the country outlines are very
large objects and are toasted out-of-line. There's been some past
discussion of automatically avoiding repeated detoastings in scenarios
like the above, but nothing's gotten to the point of acceptance yet.
Possibly you could do something to force detoasting in a subquery.
regards, tom lane
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Performance issues
2011-03-07 13:45 Performance issues Andreas Forø Tollefsen <andreasft@gmail.com>
2011-03-07 14:01 ` Re: Performance issues Kenneth Marshall <ktm@rice.edu>
2011-03-07 14:17 ` Re: Performance issues Andreas Forø Tollefsen <andreasft@gmail.com>
2011-03-07 14:22 ` Re: Performance issues Kenneth Marshall <ktm@rice.edu>
2011-03-07 14:29 ` Re: Performance issues Andreas Forø Tollefsen <andreasft@gmail.com>
2011-03-07 15:34 ` Re: Performance issues Oleg Bartunov <oleg@sai.msu.su>
2011-03-07 21:49 ` Re: Performance issues Andreas Forø Tollefsen <andreasft@gmail.com>
2011-03-08 00:38 ` Re: Performance issues Tom Lane <tgl@sss.pgh.pa.us>
@ 2011-03-08 08:42 ` Andreas Forø Tollefsen <andreasft@gmail.com>
2011-03-08 15:21 ` Re: Performance issues Andy Colson <andy@squeakycode.net>
0 siblings, 1 reply; 59+ messages in thread
From: Andreas Forø Tollefsen @ 2011-03-08 08:42 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Oleg Bartunov <oleg@sai.msu.su>; Kenneth Marshall <ktm@rice.edu>; pgsql-performance
Hi. Thanks for the comments. My data is right, and the result is exactly
what i want, but as you say i think what causes the query to be slow is the
ST_Intersection which creates the intersection between the vector grid
(fishnet) and the country polygons.
I will check with the postgis user list if they have any idea on how to
speed up this query.
Best,
Andreas
2011/3/8 Tom Lane <tgl@sss.pgh.pa.us>
> =?ISO-8859-1?Q?Andreas_For=F8_Tollefsen?= <andreasft@gmail.com> writes:
> > This is a query i am working on now. It creates an intersection of two
> > geometries. One is a grid of 0.5 x 0.5 decimal degree sized cells, while
> the
> > other is the country geometries of all countries in the world for a
> certain
> > year.
>
> Hm, are you sure your data is right? Because the actual rowcounts imply
> that each country intersects about half of the grid cells, which doesn't
> seem right.
>
> > priogrid=# EXPLAIN ANALYZE SELECT priogrid_land.gid, gwcode,
> > ST_Intersection(pri
> > ogrid_land.cell, cshapeswdate.geom) FROM priogrid_land, cshapeswdate
> WHERE
> > ST_Intersects(priogrid_land.cell, cshapeswdate.geom);
> > QUERY
> > PLAN
>
> >
> --------------------------------------------------------------------------------
> > ------------------------------------------------------------------
> > Nested Loop (cost=0.00..12644.85 rows=43351 width=87704) (actual
> > time=1.815..7
> > 074973.711 rows=130331 loops=1)
> > Join Filter: _st_intersects(priogrid_land.cell, cshapeswdate.geom)
> > -> Seq Scan on cshapeswdate (cost=0.00..14.42 rows=242 width=87248)
> > (actual
> > time=0.007..0.570 rows=242 loops=1)
> > -> Index Scan using idx_priogrid_land_cell on priogrid_land
> > (cost=0.00..7.1
> > 5 rows=1 width=456) (actual time=0.069..5.604 rows=978 loops=242)
> > Index Cond: (priogrid_land.cell && cshapeswdate.geom)
> > Total runtime: 7075188.549 ms
> > (6 rows)
>
> AFAICT, all of the runtime is going into calculating the ST_Intersects
> and/or ST_Intersection functions. The two scans are only accounting for
> perhaps 5.5 seconds, and the join infrastructure isn't going to be
> terribly expensive, so it's got to be those functions. Not knowing much
> about PostGIS, I don't know if the functions themselves can be expected
> to be really slow. If it's not them, it could be the cost of fetching
> their arguments --- in particular, I bet the country outlines are very
> large objects and are toasted out-of-line. There's been some past
> discussion of automatically avoiding repeated detoastings in scenarios
> like the above, but nothing's gotten to the point of acceptance yet.
> Possibly you could do something to force detoasting in a subquery.
>
> regards, tom lane
>
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Performance issues
2011-03-07 13:45 Performance issues Andreas Forø Tollefsen <andreasft@gmail.com>
2011-03-07 14:01 ` Re: Performance issues Kenneth Marshall <ktm@rice.edu>
2011-03-07 14:17 ` Re: Performance issues Andreas Forø Tollefsen <andreasft@gmail.com>
2011-03-07 14:22 ` Re: Performance issues Kenneth Marshall <ktm@rice.edu>
2011-03-07 14:29 ` Re: Performance issues Andreas Forø Tollefsen <andreasft@gmail.com>
2011-03-07 15:34 ` Re: Performance issues Oleg Bartunov <oleg@sai.msu.su>
2011-03-07 21:49 ` Re: Performance issues Andreas Forø Tollefsen <andreasft@gmail.com>
2011-03-08 00:38 ` Re: Performance issues Tom Lane <tgl@sss.pgh.pa.us>
2011-03-08 08:42 ` Re: Performance issues Andreas Forø Tollefsen <andreasft@gmail.com>
@ 2011-03-08 15:21 ` Andy Colson <andy@squeakycode.net>
2011-03-08 16:58 ` Re: Performance issues Andreas Forø Tollefsen <andreasft@gmail.com>
0 siblings, 1 reply; 59+ messages in thread
From: Andy Colson @ 2011-03-08 15:21 UTC (permalink / raw)
To: Andreas Forø Tollefsen <andreasft@gmail.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Oleg Bartunov <oleg@sai.msu.su>; Kenneth Marshall <ktm@rice.edu>; pgsql-performance
I have seen really complex geometries cause problems. If you have
thousands of points, when 10 would do, try ST_Simplify and see if it
doesnt speed things up.
-Andy
On 3/8/2011 2:42 AM, Andreas Forø Tollefsen wrote:
> Hi. Thanks for the comments. My data is right, and the result is exactly
> what i want, but as you say i think what causes the query to be slow is
> the ST_Intersection which creates the intersection between the vector
> grid (fishnet) and the country polygons.
> I will check with the postgis user list if they have any idea on how to
> speed up this query.
>
> Best,
> Andreas
>
> 2011/3/8 Tom Lane <tgl@sss.pgh.pa.us <mailto:tgl@sss.pgh.pa.us>>
>
> =?ISO-8859-1?Q?Andreas_For=F8_Tollefsen?= <andreasft@gmail.com
> <mailto:andreasft@gmail.com>> writes:
> > This is a query i am working on now. It creates an intersection
> of two
> > geometries. One is a grid of 0.5 x 0.5 decimal degree sized
> cells, while the
> > other is the country geometries of all countries in the world for
> a certain
> > year.
>
> Hm, are you sure your data is right? Because the actual rowcounts imply
> that each country intersects about half of the grid cells, which doesn't
> seem right.
>
> > priogrid=# EXPLAIN ANALYZE SELECT priogrid_land.gid, gwcode,
> > ST_Intersection(pri
> > ogrid_land.cell, cshapeswdate.geom) FROM priogrid_land,
> cshapeswdate WHERE
> > ST_Intersects(priogrid_land.cell, cshapeswdate.geom);
> >
> QUERY
> > PLAN
>
> >
> --------------------------------------------------------------------------------
> > ------------------------------------------------------------------
> > Nested Loop (cost=0.00..12644.85 rows=43351 width=87704) (actual
> > time=1.815..7
> > 074973.711 rows=130331 loops=1)
> > Join Filter: _st_intersects(priogrid_land.cell, cshapeswdate.geom)
> > -> Seq Scan on cshapeswdate (cost=0.00..14.42 rows=242
> width=87248)
> > (actual
> > time=0.007..0.570 rows=242 loops=1)
> > -> Index Scan using idx_priogrid_land_cell on priogrid_land
> > (cost=0.00..7.1
> > 5 rows=1 width=456) (actual time=0.069..5.604 rows=978 loops=242)
> > Index Cond: (priogrid_land.cell && cshapeswdate.geom)
> > Total runtime: 7075188.549 ms
> > (6 rows)
>
> AFAICT, all of the runtime is going into calculating the ST_Intersects
> and/or ST_Intersection functions. The two scans are only accounting for
> perhaps 5.5 seconds, and the join infrastructure isn't going to be
> terribly expensive, so it's got to be those functions. Not knowing much
> about PostGIS, I don't know if the functions themselves can be expected
> to be really slow. If it's not them, it could be the cost of fetching
> their arguments --- in particular, I bet the country outlines are very
> large objects and are toasted out-of-line. There's been some past
> discussion of automatically avoiding repeated detoastings in scenarios
> like the above, but nothing's gotten to the point of acceptance yet.
> Possibly you could do something to force detoasting in a subquery.
>
> regards, tom lane
>
>
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Performance issues
2011-03-07 13:45 Performance issues Andreas Forø Tollefsen <andreasft@gmail.com>
2011-03-07 14:01 ` Re: Performance issues Kenneth Marshall <ktm@rice.edu>
2011-03-07 14:17 ` Re: Performance issues Andreas Forø Tollefsen <andreasft@gmail.com>
2011-03-07 14:22 ` Re: Performance issues Kenneth Marshall <ktm@rice.edu>
2011-03-07 14:29 ` Re: Performance issues Andreas Forø Tollefsen <andreasft@gmail.com>
2011-03-07 15:34 ` Re: Performance issues Oleg Bartunov <oleg@sai.msu.su>
2011-03-07 21:49 ` Re: Performance issues Andreas Forø Tollefsen <andreasft@gmail.com>
2011-03-08 00:38 ` Re: Performance issues Tom Lane <tgl@sss.pgh.pa.us>
2011-03-08 08:42 ` Re: Performance issues Andreas Forø Tollefsen <andreasft@gmail.com>
2011-03-08 15:21 ` Re: Performance issues Andy Colson <andy@squeakycode.net>
@ 2011-03-08 16:58 ` Andreas Forø Tollefsen <andreasft@gmail.com>
2011-03-08 17:00 ` Re: Performance issues Andreas Forø Tollefsen <andreasft@gmail.com>
2011-03-08 19:13 ` Re: Performance issues Andy Colson <andy@squeakycode.net>
0 siblings, 2 replies; 59+ messages in thread
From: Andreas Forø Tollefsen @ 2011-03-08 16:58 UTC (permalink / raw)
To: Andy Colson <andy@squeakycode.net>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Oleg Bartunov <oleg@sai.msu.su>; Kenneth Marshall <ktm@rice.edu>; pgsql-performance
Andy. Thanks. That is a great tips. I tried it but i get the error:
NOTICE: ptarray_simplify returned a <2 pts array.
Query:
SELECT ST_Intersection(priogrid_land.cell,
ST_Simplify(cshapeswdate.geom,0.1)) AS geom,
priogrid_land.gid AS divider, gwcode, gwsyear, gweyear, startdate, enddate,
capname, caplong, caplat, col, row, xcoord, ycoord
FROM priogrid_land, cshapeswdate WHERE ST_Intersects(priogrid_land.cell,
ST_Simplify(cshapeswdate.geom,0.1)) AND cshapeswdate.gwsyear <=1946 AND
cshapeswdate.gweyear >=1946 AND cshapeswdate.startdate <= '1946/1/1';
2011/3/8 Andy Colson <andy@squeakycode.net>
> I have seen really complex geometries cause problems. If you have
> thousands of points, when 10 would do, try ST_Simplify and see if it doesnt
> speed things up.
>
> -Andy
>
>
>
> On 3/8/2011 2:42 AM, Andreas Forř Tollefsen wrote:
>
>> Hi. Thanks for the comments. My data is right, and the result is exactly
>> what i want, but as you say i think what causes the query to be slow is
>> the ST_Intersection which creates the intersection between the vector
>> grid (fishnet) and the country polygons.
>> I will check with the postgis user list if they have any idea on how to
>> speed up this query.
>>
>> Best,
>> Andreas
>>
>> 2011/3/8 Tom Lane <tgl@sss.pgh.pa.us <mailto:tgl@sss.pgh.pa.us>>
>>
>>
>> =?ISO-8859-1?Q?Andreas_For=F8_Tollefsen?= <andreasft@gmail.com
>> <mailto:andreasft@gmail.com>> writes:
>> > This is a query i am working on now. It creates an intersection
>> of two
>> > geometries. One is a grid of 0.5 x 0.5 decimal degree sized
>> cells, while the
>> > other is the country geometries of all countries in the world for
>> a certain
>> > year.
>>
>> Hm, are you sure your data is right? Because the actual rowcounts
>> imply
>> that each country intersects about half of the grid cells, which
>> doesn't
>> seem right.
>>
>> > priogrid=# EXPLAIN ANALYZE SELECT priogrid_land.gid, gwcode,
>> > ST_Intersection(pri
>> > ogrid_land.cell, cshapeswdate.geom) FROM priogrid_land,
>> cshapeswdate WHERE
>> > ST_Intersects(priogrid_land.cell, cshapeswdate.geom);
>> >
>> QUERY
>> > PLAN
>>
>> >
>>
>> --------------------------------------------------------------------------------
>> > ------------------------------------------------------------------
>> > Nested Loop (cost=0.00..12644.85 rows=43351 width=87704) (actual
>> > time=1.815..7
>> > 074973.711 rows=130331 loops=1)
>> > Join Filter: _st_intersects(priogrid_land.cell,
>> cshapeswdate.geom)
>> > -> Seq Scan on cshapeswdate (cost=0.00..14.42 rows=242
>> width=87248)
>> > (actual
>> > time=0.007..0.570 rows=242 loops=1)
>> > -> Index Scan using idx_priogrid_land_cell on priogrid_land
>> > (cost=0.00..7.1
>> > 5 rows=1 width=456) (actual time=0.069..5.604 rows=978 loops=242)
>> > Index Cond: (priogrid_land.cell && cshapeswdate.geom)
>> > Total runtime: 7075188.549 ms
>> > (6 rows)
>>
>> AFAICT, all of the runtime is going into calculating the ST_Intersects
>> and/or ST_Intersection functions. The two scans are only accounting
>> for
>> perhaps 5.5 seconds, and the join infrastructure isn't going to be
>> terribly expensive, so it's got to be those functions. Not knowing
>> much
>> about PostGIS, I don't know if the functions themselves can be expected
>> to be really slow. If it's not them, it could be the cost of fetching
>> their arguments --- in particular, I bet the country outlines are very
>> large objects and are toasted out-of-line. There's been some past
>> discussion of automatically avoiding repeated detoastings in scenarios
>> like the above, but nothing's gotten to the point of acceptance yet.
>> Possibly you could do something to force detoasting in a subquery.
>>
>> regards, tom lane
>>
>>
>>
>
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Performance issues
2011-03-07 13:45 Performance issues Andreas Forø Tollefsen <andreasft@gmail.com>
2011-03-07 14:01 ` Re: Performance issues Kenneth Marshall <ktm@rice.edu>
2011-03-07 14:17 ` Re: Performance issues Andreas Forø Tollefsen <andreasft@gmail.com>
2011-03-07 14:22 ` Re: Performance issues Kenneth Marshall <ktm@rice.edu>
2011-03-07 14:29 ` Re: Performance issues Andreas Forø Tollefsen <andreasft@gmail.com>
2011-03-07 15:34 ` Re: Performance issues Oleg Bartunov <oleg@sai.msu.su>
2011-03-07 21:49 ` Re: Performance issues Andreas Forø Tollefsen <andreasft@gmail.com>
2011-03-08 00:38 ` Re: Performance issues Tom Lane <tgl@sss.pgh.pa.us>
2011-03-08 08:42 ` Re: Performance issues Andreas Forø Tollefsen <andreasft@gmail.com>
2011-03-08 15:21 ` Re: Performance issues Andy Colson <andy@squeakycode.net>
2011-03-08 16:58 ` Re: Performance issues Andreas Forø Tollefsen <andreasft@gmail.com>
@ 2011-03-08 17:00 ` Andreas Forø Tollefsen <andreasft@gmail.com>
1 sibling, 0 replies; 59+ messages in thread
From: Andreas Forø Tollefsen @ 2011-03-08 17:00 UTC (permalink / raw)
To: Andy Colson <andy@squeakycode.net>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Oleg Bartunov <oleg@sai.msu.su>; Kenneth Marshall <ktm@rice.edu>; pgsql-performance
Forgot to mention that the query terminates the connection because of a
crash of server process.
2011/3/8 Andreas Forø Tollefsen <andreasft@gmail.com>
> Andy. Thanks. That is a great tips. I tried it but i get the error:
> NOTICE: ptarray_simplify returned a <2 pts array.
>
> Query:
> SELECT ST_Intersection(priogrid_land.cell,
> ST_Simplify(cshapeswdate.geom,0.1)) AS geom,
> priogrid_land.gid AS divider, gwcode, gwsyear, gweyear, startdate, enddate,
> capname, caplong, caplat, col, row, xcoord, ycoord
> FROM priogrid_land, cshapeswdate WHERE ST_Intersects(priogrid_land.cell,
> ST_Simplify(cshapeswdate.geom,0.1)) AND cshapeswdate.gwsyear <=1946 AND
> cshapeswdate.gweyear >=1946 AND cshapeswdate.startdate <= '1946/1/1';
>
>
> 2011/3/8 Andy Colson <andy@squeakycode.net>
>
> I have seen really complex geometries cause problems. If you have
>> thousands of points, when 10 would do, try ST_Simplify and see if it doesnt
>> speed things up.
>>
>> -Andy
>>
>>
>>
>> On 3/8/2011 2:42 AM, Andreas Forř Tollefsen wrote:
>>
>>> Hi. Thanks for the comments. My data is right, and the result is exactly
>>> what i want, but as you say i think what causes the query to be slow is
>>> the ST_Intersection which creates the intersection between the vector
>>> grid (fishnet) and the country polygons.
>>> I will check with the postgis user list if they have any idea on how to
>>> speed up this query.
>>>
>>> Best,
>>> Andreas
>>>
>>> 2011/3/8 Tom Lane <tgl@sss.pgh.pa.us <mailto:tgl@sss.pgh.pa.us>>
>>>
>>>
>>> =?ISO-8859-1?Q?Andreas_For=F8_Tollefsen?= <andreasft@gmail.com
>>> <mailto:andreasft@gmail.com>> writes:
>>> > This is a query i am working on now. It creates an intersection
>>> of two
>>> > geometries. One is a grid of 0.5 x 0.5 decimal degree sized
>>> cells, while the
>>> > other is the country geometries of all countries in the world for
>>> a certain
>>> > year.
>>>
>>> Hm, are you sure your data is right? Because the actual rowcounts
>>> imply
>>> that each country intersects about half of the grid cells, which
>>> doesn't
>>> seem right.
>>>
>>> > priogrid=# EXPLAIN ANALYZE SELECT priogrid_land.gid, gwcode,
>>> > ST_Intersection(pri
>>> > ogrid_land.cell, cshapeswdate.geom) FROM priogrid_land,
>>> cshapeswdate WHERE
>>> > ST_Intersects(priogrid_land.cell, cshapeswdate.geom);
>>> >
>>> QUERY
>>> > PLAN
>>>
>>> >
>>>
>>> --------------------------------------------------------------------------------
>>> > ------------------------------------------------------------------
>>> > Nested Loop (cost=0.00..12644.85 rows=43351 width=87704) (actual
>>> > time=1.815..7
>>> > 074973.711 rows=130331 loops=1)
>>> > Join Filter: _st_intersects(priogrid_land.cell,
>>> cshapeswdate.geom)
>>> > -> Seq Scan on cshapeswdate (cost=0.00..14.42 rows=242
>>> width=87248)
>>> > (actual
>>> > time=0.007..0.570 rows=242 loops=1)
>>> > -> Index Scan using idx_priogrid_land_cell on priogrid_land
>>> > (cost=0.00..7.1
>>> > 5 rows=1 width=456) (actual time=0.069..5.604 rows=978 loops=242)
>>> > Index Cond: (priogrid_land.cell && cshapeswdate.geom)
>>> > Total runtime: 7075188.549 ms
>>> > (6 rows)
>>>
>>> AFAICT, all of the runtime is going into calculating the ST_Intersects
>>> and/or ST_Intersection functions. The two scans are only accounting
>>> for
>>> perhaps 5.5 seconds, and the join infrastructure isn't going to be
>>> terribly expensive, so it's got to be those functions. Not knowing
>>> much
>>> about PostGIS, I don't know if the functions themselves can be
>>> expected
>>> to be really slow. If it's not them, it could be the cost of fetching
>>> their arguments --- in particular, I bet the country outlines are very
>>> large objects and are toasted out-of-line. There's been some past
>>> discussion of automatically avoiding repeated detoastings in scenarios
>>> like the above, but nothing's gotten to the point of acceptance yet.
>>> Possibly you could do something to force detoasting in a subquery.
>>>
>>> regards, tom lane
>>>
>>>
>>>
>>
>
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Performance issues
2011-03-07 13:45 Performance issues Andreas Forø Tollefsen <andreasft@gmail.com>
2011-03-07 14:01 ` Re: Performance issues Kenneth Marshall <ktm@rice.edu>
2011-03-07 14:17 ` Re: Performance issues Andreas Forø Tollefsen <andreasft@gmail.com>
2011-03-07 14:22 ` Re: Performance issues Kenneth Marshall <ktm@rice.edu>
2011-03-07 14:29 ` Re: Performance issues Andreas Forø Tollefsen <andreasft@gmail.com>
2011-03-07 15:34 ` Re: Performance issues Oleg Bartunov <oleg@sai.msu.su>
2011-03-07 21:49 ` Re: Performance issues Andreas Forø Tollefsen <andreasft@gmail.com>
2011-03-08 00:38 ` Re: Performance issues Tom Lane <tgl@sss.pgh.pa.us>
2011-03-08 08:42 ` Re: Performance issues Andreas Forø Tollefsen <andreasft@gmail.com>
2011-03-08 15:21 ` Re: Performance issues Andy Colson <andy@squeakycode.net>
2011-03-08 16:58 ` Re: Performance issues Andreas Forø Tollefsen <andreasft@gmail.com>
@ 2011-03-08 19:13 ` Andy Colson <andy@squeakycode.net>
1 sibling, 0 replies; 59+ messages in thread
From: Andy Colson @ 2011-03-08 19:13 UTC (permalink / raw)
To: Andreas Forø Tollefsen <andreasft@gmail.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Oleg Bartunov <oleg@sai.msu.su>; Kenneth Marshall <ktm@rice.edu>; pgsql-performance
On 3/8/2011 10:58 AM, Andreas Forø Tollefsen wrote:
> Andy. Thanks. That is a great tips. I tried it but i get the error:
> NOTICE: ptarray_simplify returned a <2 pts array.
>
> Query:
> SELECT ST_Intersection(priogrid_land.cell,
> ST_Simplify(cshapeswdate.geom,0.1)) AS geom,
> priogrid_land.gid AS divider, gwcode, gwsyear, gweyear, startdate,
> enddate, capname, caplong, caplat, col, row, xcoord, ycoord
> FROM priogrid_land, cshapeswdate WHERE ST_Intersects(priogrid_land.cell,
> ST_Simplify(cshapeswdate.geom,0.1)) AND cshapeswdate.gwsyear <=1946 AND
> cshapeswdate.gweyear >=1946 AND cshapeswdate.startdate <= '1946/1/1';
>
>
> 2011/3/8 Andy Colson <andy@squeakycode.net <mailto:andy@squeakycode.net>>
>
> I have seen really complex geometries cause problems. If you have
> thousands of points, when 10 would do, try ST_Simplify and see if it
> doesnt speed things up.
>
> -Andy
>
>
>
> On 3/8/2011 2:42 AM, Andreas Forř Tollefsen wrote:
>
> Hi. Thanks for the comments. My data is right, and the result is
> exactly
> what i want, but as you say i think what causes the query to be
> slow is
> the ST_Intersection which creates the intersection between the
> vector
> grid (fishnet) and the country polygons.
> I will check with the postgis user list if they have any idea on
> how to
> speed up this query.
>
> Best,
> Andreas
>
> 2011/3/8 Tom Lane <tgl@sss.pgh.pa.us <mailto:tgl@sss.pgh.pa.us>
> <mailto:tgl@sss.pgh.pa.us <mailto:tgl@sss.pgh.pa.us>>>
>
>
> =?ISO-8859-1?Q?Andreas_For=F8_Tollefsen?=
> <andreasft@gmail.com <mailto:andreasft@gmail.com>
> <mailto:andreasft@gmail.com <mailto:andreasft@gmail.com>>> writes:
> > This is a query i am working on now. It creates an intersection
> of two
> > geometries. One is a grid of 0.5 x 0.5 decimal degree sized
> cells, while the
> > other is the country geometries of all countries in the world for
> a certain
> > year.
>
> Hm, are you sure your data is right? Because the actual
> rowcounts imply
> that each country intersects about half of the grid cells,
> which doesn't
> seem right.
>
> > priogrid=# EXPLAIN ANALYZE SELECT priogrid_land.gid, gwcode,
> > ST_Intersection(pri
> > ogrid_land.cell, cshapeswdate.geom) FROM priogrid_land,
> cshapeswdate WHERE
> > ST_Intersects(priogrid_land.cell, cshapeswdate.geom);
> >
> QUERY
> > PLAN
>
> >
>
> --------------------------------------------------------------------------------
> >
> ------------------------------------------------------------------
> > Nested Loop (cost=0.00..12644.85 rows=43351 width=87704)
> (actual
> > time=1.815..7
> > 074973.711 rows=130331 loops=1)
> > Join Filter: _st_intersects(priogrid_land.cell,
> cshapeswdate.geom)
> > -> Seq Scan on cshapeswdate (cost=0.00..14.42 rows=242
> width=87248)
> > (actual
> > time=0.007..0.570 rows=242 loops=1)
> > -> Index Scan using idx_priogrid_land_cell on priogrid_land
> > (cost=0.00..7.1
> > 5 rows=1 width=456) (actual time=0.069..5.604 rows=978 loops=242)
> > Index Cond: (priogrid_land.cell && cshapeswdate.geom)
> > Total runtime: 7075188.549 ms
> > (6 rows)
>
> AFAICT, all of the runtime is going into calculating the
> ST_Intersects
> and/or ST_Intersection functions. The two scans are only
> accounting for
> perhaps 5.5 seconds, and the join infrastructure isn't going
> to be
> terribly expensive, so it's got to be those functions. Not
> knowing much
> about PostGIS, I don't know if the functions themselves can
> be expected
> to be really slow. If it's not them, it could be the cost
> of fetching
> their arguments --- in particular, I bet the country
> outlines are very
> large objects and are toasted out-of-line. There's been
> some past
> discussion of automatically avoiding repeated detoastings in
> scenarios
> like the above, but nothing's gotten to the point of
> acceptance yet.
> Possibly you could do something to force detoasting in a
> subquery.
>
> regards, tom lane
>
>
>
>
ew... thats not good. Seems like it simplified it down to a single
point? (not 100% sure that's what the error means, just a guess)
Try getting some info about it:
select
ST_Npoints(geom) As before,
ST_NPoints(ST_Simplify(geom,0.1)) as after
from cshapeswdate
Also try things like ST_IsSimple ST_IsValid. I seem to recall sometimes
needing ST_Points or st_NumPoints instead of ST_Npoints.
-Andy
^ permalink raw reply [nested|flat] 59+ messages in thread
* Performance issues
@ 2015-03-13 11:44 Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 12:58 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
0 siblings, 1 reply; 59+ messages in thread
From: Vivekanand Joshi @ 2015-03-13 11:44 UTC (permalink / raw)
To: pgsql-performance
Hi Team,
I am a novice to this territory. We are trying to migrate few jasper
reports from Netezza to PostgreSQL.
I have one report ready with me but queries are taking too much time. To be
honest, it is not giving any result most of the time.
The same query in Netezza is running in less than 2-3 seconds.
========================================================================================================
This is the query :
SELECT
COUNT(DISTINCT TARGET_ID)
FROM
S_V_F_PROMOTION_HISTORY_EMAIL PH
INNER JOIN S_V_D_CAMPAIGN_HIERARCHY CH
ON PH.TOUCHPOINT_EXECUTION_ID =
CH.TOUCHPOINT_EXECUTION_ID
WHERE
1=1
AND SEND_DT >= '2014-03-13'
AND SEND_DT <= '2015-03-14'
Statistics:
Select Count(1) from S_V_F_PROMOTION_HISTORY_EMAIL
4559289
Time: 16781.409 ms
Select count(1) from S_V_D_CAMPAIGN_HIERARCHY;
count
-------
45360
(1 row)
Time: 467869.185 ms
==================================================================
EXPLAIN PLAN FOR QUERY:
"Aggregate (cost=356422.36..356422.37 rows=1 width=8)"
" Output: count(DISTINCT base.target_id)"
" -> Nested Loop (cost=68762.23..356422.36 rows=1 width=8)"
" Output: base.target_id"
" Join Filter: (base.touchpoint_execution_id =
tp_exec.touchpoint_execution_id)"
" -> Nested Loop (cost=33927.73..38232.16 rows=1 width=894)"
" Output: camp.campaign_id, camp.campaign_name,
camp.initiative, camp.objective, camp.category_id,
"CATEGORY".category_name, camp_exec.campaign_execution_id,
camp_exec.campaign_execution_name, camp_exec.group_id, grup.group_name,
camp_exec.star (...)"
" Join Filter: (tp_exec.touchpoint_execution_id =
valid_executions.touchpoint_execution_id)"
" CTE valid_executions"
" -> Merge Join (cost=30420.45..31971.94 rows=1 width=8)"
" Output:
s_f_touchpoint_execution_status_history_2.touchpoint_execution_id"
" Merge Cond:
((s_f_touchpoint_execution_status_history_2.touchpoint_execution_id =
s_f_touchpoint_execution_status_history_1_1.touchpoint_execution_id) AND
(s_f_touchpoint_execution_status_history_2.creation_dt =
(max(s_f_touchpoint_ex (...)"
" -> Sort (cost=17196.30..17539.17 rows=137149
width=16)"
" Output:
s_f_touchpoint_execution_status_history_2.touchpoint_execution_id,
s_f_touchpoint_execution_status_history_2.creation_dt"
" Sort Key:
s_f_touchpoint_execution_status_history_2.touchpoint_execution_id,
s_f_touchpoint_execution_status_history_2.creation_dt"
" -> Seq Scan on
public.s_f_touchpoint_execution_status_history
s_f_touchpoint_execution_status_history_2 (cost=0.00..5493.80 rows=137149
width=16)"
" Output:
s_f_touchpoint_execution_status_history_2.touchpoint_execution_id,
s_f_touchpoint_execution_status_history_2.creation_dt"
" Filter:
(s_f_touchpoint_execution_status_history_2.touchpoint_execution_status_type_id
= ANY ('{3,4}'::integer[]))"
" -> Sort (cost=13224.15..13398.43 rows=69715
width=16)"
" Output:
s_f_touchpoint_execution_status_history_1_1.touchpoint_execution_id,
(max(s_f_touchpoint_execution_status_history_1_1.creation_dt))"
" Sort Key:
s_f_touchpoint_execution_status_history_1_1.touchpoint_execution_id,
(max(s_f_touchpoint_execution_status_history_1_1.creation_dt))"
" -> HashAggregate (cost=6221.56..6918.71
rows=69715 width=16)"
" Output:
s_f_touchpoint_execution_status_history_1_1.touchpoint_execution_id,
max(s_f_touchpoint_execution_status_history_1_1.creation_dt)"
" Group Key:
s_f_touchpoint_execution_status_history_1_1.touchpoint_execution_id"
" -> Seq Scan on
public.s_f_touchpoint_execution_status_history
s_f_touchpoint_execution_status_history_1_1 (cost=0.00..4766.04
rows=291104 width=16)"
" Output:
s_f_touchpoint_execution_status_history_1_1.touchpoint_execution_id,
s_f_touchpoint_execution_status_history_1_1.touchpoint_execution_status_type_id,
s_f_touchpoint_execution_status_history_1_1.status_message (...)"
" -> Nested Loop Left Join (cost=1955.80..6260.19 rows=1
width=894)"
" Output: tp_exec.touchpoint_execution_id,
tp_exec.start_dt, tp_exec.message_type_id, tp_exec.content_id,
wave_exec.wave_execution_id, wave_exec.wave_execution_name,
wave_exec.start_dt, camp_exec.campaign_execution_id, camp_exec.campaign_
(...)"
" -> Nested Loop (cost=1955.67..6260.04 rows=1
width=776)"
" Output: tp_exec.touchpoint_execution_id,
tp_exec.start_dt, tp_exec.message_type_id, tp_exec.content_id,
wave_exec.wave_execution_id, wave_exec.wave_execution_name,
wave_exec.start_dt, camp_exec.campaign_execution_id, camp_exec.cam (...)"
" -> Nested Loop Left Join
(cost=1955.54..6259.87 rows=1 width=658)"
" Output: tp_exec.touchpoint_execution_id,
tp_exec.start_dt, tp_exec.message_type_id, tp_exec.content_id,
wave_exec.wave_execution_id, wave_exec.wave_execution_name,
wave_exec.start_dt, camp_exec.campaign_execution_id, camp_ex (...)"
" -> Nested Loop Left Join
(cost=1955.40..6259.71 rows=1 width=340)"
" Output:
tp_exec.touchpoint_execution_id, tp_exec.start_dt, tp_exec.message_type_id,
tp_exec.content_id, wave_exec.wave_execution_id,
wave_exec.wave_execution_name, wave_exec.start_dt,
camp_exec.campaign_execution_id, c (...)"
" -> Nested Loop Left Join
(cost=1955.27..6259.55 rows=1 width=222)"
" Output:
tp_exec.touchpoint_execution_id, tp_exec.start_dt, tp_exec.message_type_id,
tp_exec.content_id, wave_exec.wave_execution_id,
wave_exec.wave_execution_name, wave_exec.start_dt,
camp_exec.campaign_execution (...)"
" -> Nested Loop
(cost=1954.99..6259.24 rows=1 width=197)"
" Output:
tp_exec.touchpoint_execution_id, tp_exec.start_dt, tp_exec.message_type_id,
tp_exec.content_id, wave_exec.wave_execution_id,
wave_exec.wave_execution_name, wave_exec.start_dt, camp_exec.campaign_exe
(...)"
" -> Nested Loop
(cost=1954.71..6258.92 rows=1 width=173)"
" Output:
tp_exec.touchpoint_execution_id, tp_exec.start_dt, tp_exec.message_type_id,
tp_exec.content_id, wave_exec.wave_execution_id,
wave_exec.wave_execution_name, wave_exec.start_dt, camp_exec.campai (...)"
" Join Filter:
(camp_exec.campaign_id = wave.campaign_id)"
" -> Nested Loop
(cost=1954.42..6254.67 rows=13 width=167)"
" Output:
tp_exec.touchpoint_execution_id, tp_exec.start_dt, tp_exec.message_type_id,
tp_exec.content_id, wave_exec.wave_execution_id,
wave_exec.wave_execution_name, wave_exec.start_dt, wave_exec. (...)"
" -> Hash
Join (cost=1954.13..6249.67 rows=13 width=108)"
"
Output: tp_exec.touchpoint_execution_id, tp_exec.start_dt,
tp_exec.message_type_id, tp_exec.content_id, wave_exec.wave_execution_id,
wave_exec.wave_execution_name, wave_exec.start_dt, wave (...)"
" Hash
Cond: ((tp_exec.touchpoint_id = tp.touchpoint_id) AND (wave_exec.wave_id =
tp.wave_id))"
" ->
Hash Join (cost=1576.83..4595.51 rows=72956 width=90)"
"
Output: tp_exec.touchpoint_execution_id,
tp_exec.start_dt, tp_exec.message_type_id, tp_exec.content_id,
tp_exec.touchpoint_id, wave_exec.wave_execution_id,
wave_exec.wave_execution_n (...)"
"
Hash Cond:
(tp_exec.wave_execution_id = wave_exec.wave_execution_id)"
"
-> Seq Scan on public.s_d_touchpoint_execution tp_exec
(cost=0.00..1559.56 rows=72956 width=42)"
"
Output: tp_exec.touchpoint_execution_id, tp_exec.wave_execution_id,
tp_exec.touchpoint_id, tp_exec.channel_type_id, tp_exec.content_id,
tp_exec.message_type_id, tp_exec.start_d (...)"
"
-> Hash (cost=1001.37..1001.37 rows=46037 width=56)"
"
Output: wave_exec.wave_execution_id, wave_exec.wave_execution_name,
wave_exec.start_dt, wave_exec.campaign_execution_id, wave_exec.wave_id"
"
-> Seq Scan on public.s_d_wave_execution wave_exec (cost=0.00..1001.37
rows=46037 width=56)"
"
Output: wave_exec.wave_execution_id, wave_exec.wave_execution_name,
wave_exec.start_dt, wave_exec.campaign_execution_id, wave_exec.wave_id"
" ->
Hash (cost=212.72..212.72 rows=10972 width=26)"
"
Output: tp.touchpoint_id, tp.touchpoint_name, tp.wave_id,
tp.channel_type_id"
"
-> Seq Scan on public.s_d_touchpoint tp (cost=0.00..212.72 rows=10972
width=26)"
"
Output: tp.touchpoint_id, tp.touchpoint_name, tp.wave_id,
tp.channel_type_id"
" -> Index
Scan using s_d_campaign_execution_idx on public.s_d_campaign_execution
camp_exec (cost=0.29..0.37 rows=1 width=67)"
"
Output: camp_exec.campaign_execution_id, camp_exec.campaign_id,
camp_exec.group_id, camp_exec.campaign_execution_name, camp_exec.start_dt,
camp_exec.creation_dt"
" Index
Cond: (camp_exec.campaign_execution_id = wave_exec.campaign_execution_id)"
" -> Index Scan
using s_d_wave_pkey on public.s_d_wave wave (cost=0.29..0.31 rows=1
width=22)"
" Output:
wave.wave_id, wave.campaign_id, wave.wave_name, wave.creation_dt,
wave.modified_dt"
" Index Cond:
(wave.wave_id = wave_exec.wave_id)"
" -> Index Scan using
s_d_campaign_pkey on public.s_d_campaign camp (cost=0.29..0.32 rows=1
width=40)"
" Output:
camp.campaign_id, camp.campaign_name, camp.objective, camp.initiative,
camp.category_id, camp.creation_dt, camp.modified_dt"
" Index Cond:
(camp.campaign_id = camp_exec.campaign_id)"
" -> Index Scan using
s_d_content_pkey on public.s_d_content content (cost=0.28..0.30 rows=1
width=33)"
" Output:
content.content_id, content.content_name, content.creation_dt,
content.channel_type_id, content.modified_dt"
" Index Cond:
(tp_exec.content_id = content.content_id)"
" -> Index Scan using
s_d_message_type_pkey on public.s_d_message_type message_type
(cost=0.13..0.15 rows=1 width=120)"
" Output:
message_type.message_type_id, message_type.message_type_name,
message_type.creation_dt, message_type.modified_dt"
" Index Cond:
(tp_exec.message_type_id = message_type.message_type_id)"
" -> Index Scan using s_d_group_pkey on
public.s_d_group grup (cost=0.13..0.15 rows=1 width=320)"
" Output: grup.group_id,
grup.group_name, grup.creation_dt, grup.modified_dt"
" Index Cond: (camp_exec.group_id =
grup.group_id)"
" -> Index Scan using d_channel_pk on
public.s_d_channel_type channel (cost=0.13..0.15 rows=1 width=120)"
" Output: channel.channel_type_id,
channel.channel_type_name"
" Index Cond: (channel.channel_type_id =
tp.channel_type_id)"
" -> Index Scan using s_d_category_pkey on
public.s_d_category "CATEGORY" (cost=0.13..0.15 rows=1 width=120)"
" Output: "CATEGORY".category_id,
"CATEGORY".category_name, "CATEGORY".creation_dt, "CATEGORY".modified_dt"
" Index Cond: (camp.category_id =
"CATEGORY".category_id)"
" -> CTE Scan on valid_executions (cost=0.00..0.02 rows=1
width=8)"
" Output: valid_executions.touchpoint_execution_id"
" -> Nested Loop Left Join (cost=34834.49..318190.14 rows=2
width=148)"
" Output: base.promo_hist_id, base.audience_member_id,
base.target_id, base.touchpoint_execution_id, base.contact_group_id,
base.content_version_execution_id, base.sent_ind, CASE WHEN
(email.sbounce_ind IS NOT NULL) THEN (email.sbounce_ind)::in (...)"
" CTE valid_executions"
" -> Nested Loop (cost=33089.13..34834.20 rows=1 width=8)"
" Output:
s_f_touchpoint_execution_status_history.touchpoint_execution_id"
" -> Nested Loop (cost=33088.84..34833.88 rows=1
width=16)"
" Output:
s_f_touchpoint_execution_status_history.touchpoint_execution_id,
tpe.touchpoint_id"
" -> Unique (cost=33088.42..34825.42 rows=1
width=8)"
" Output:
s_f_touchpoint_execution_status_history.touchpoint_execution_id"
" -> Merge Join (cost=33088.42..34825.42
rows=1 width=8)"
" Output:
s_f_touchpoint_execution_status_history.touchpoint_execution_id"
" Merge Cond:
((s_f_touchpoint_execution_status_history.touchpoint_execution_id =
s_f_touchpoint_execution_status_history_1.touchpoint_execution_id) AND
(s_f_touchpoint_execution_status_history.creation_dt = (max(s_f_t (...)"
" -> Sort (cost=19864.28..20268.98
rows=161883 width=16)"
" Output:
s_f_touchpoint_execution_status_history.touchpoint_execution_id,
s_f_touchpoint_execution_status_history.creation_dt"
" Sort Key:
s_f_touchpoint_execution_status_history.touchpoint_execution_id,
s_f_touchpoint_execution_status_history.creation_dt"
" -> Seq Scan on
public.s_f_touchpoint_execution_status_history (cost=0.00..5857.68
rows=161883 width=16)"
" Output:
s_f_touchpoint_execution_status_history.touchpoint_execution_id,
s_f_touchpoint_execution_status_history.creation_dt"
" Filter:
(s_f_touchpoint_execution_status_history.touchpoint_execution_status_type_id
= ANY ('{3,4,6}'::integer[]))"
" -> Sort (cost=13224.15..13398.43
rows=69715 width=16)"
" Output:
s_f_touchpoint_execution_status_history_1.touchpoint_execution_id,
(max(s_f_touchpoint_execution_status_history_1.creation_dt))"
" Sort Key:
s_f_touchpoint_execution_status_history_1.touchpoint_execution_id,
(max(s_f_touchpoint_execution_status_history_1.creation_dt))"
" -> HashAggregate
(cost=6221.56..6918.71 rows=69715 width=16)"
" Output:
s_f_touchpoint_execution_status_history_1.touchpoint_execution_id,
max(s_f_touchpoint_execution_status_history_1.creation_dt)"
" Group Key:
s_f_touchpoint_execution_status_history_1.touchpoint_execution_id"
" -> Seq Scan on
public.s_f_touchpoint_execution_status_history
s_f_touchpoint_execution_status_history_1 (cost=0.00..4766.04 rows=291104
width=16)"
" Output:
s_f_touchpoint_execution_status_history_1.touchpoint_execution_id,
s_f_touchpoint_execution_status_history_1.touchpoint_execution_status_type_id,
s_f_touchpoint_execution_status_history_1.st (...)"
" -> Index Scan using
s_d_touchpoint_execution_pkey on public.s_d_touchpoint_execution tpe
(cost=0.42..8.44 rows=1 width=16)"
" Output: tpe.touchpoint_execution_id,
tpe.wave_execution_id, tpe.touchpoint_id, tpe.channel_type_id,
tpe.content_id, tpe.message_type_id, tpe.start_dt, tpe.creation_dt"
" Index Cond: (tpe.touchpoint_execution_id
= s_f_touchpoint_execution_status_history.touchpoint_execution_id)"
" -> Index Only Scan using s_d_touchpoint_pkey on
public.s_d_touchpoint tp_1 (cost=0.29..0.32 rows=1 width=8)"
" Output: tp_1.touchpoint_id,
tp_1.channel_type_id"
" Index Cond: ((tp_1.touchpoint_id =
tpe.touchpoint_id) AND (tp_1.channel_type_id = 1))"
" -> Nested Loop (cost=0.00..283350.22 rows=2 width=74)"
" Output: base.promo_hist_id, base.audience_member_id,
base.target_id, base.touchpoint_execution_id, base.contact_group_id,
base.content_version_execution_id, base.sent_ind, base.send_dt,
base.creation_dt, base.modified_dt"
" Join Filter: (base.touchpoint_execution_id =
valid_executions_1.touchpoint_execution_id)"
" -> CTE Scan on valid_executions valid_executions_1
(cost=0.00..0.02 rows=1 width=8)"
" Output:
valid_executions_1.touchpoint_execution_id"
" -> Seq Scan on public.s_f_promotion_history base
(cost=0.00..283334.00 rows=1296 width=74)"
" Output: base.promo_hist_id, base.target_id,
base.audience_member_id, base.touchpoint_execution_id,
base.contact_group_id, base.content_version_execution_id, base.sent_ind,
base.send_dt, base.creation_dt, base.modified_dt"
" Filter: ((base.send_dt >= '2014-03-13
00:00:00'::timestamp without time zone) AND (base.send_dt <= '2015-03-14
00:00:00'::timestamp without time zone))"
" -> Index Scan using s_f_promotion_history_email_pk1 on
public.s_f_promotion_history_email email (cost=0.29..2.83 rows=1 width=90)"
" Output: email.promo_hist_id, email.target_id,
email.audience_member_id, email.touchpoint_execution_id,
email.contact_group_id, email.sbounce_ind, email.hbounce_ind,
email.opened_ind, email.clicked_ind, email.unsubscribe_ind, email.unsub
(...)"
" Index Cond: (base.promo_hist_id = email.promo_hist_id)"
" Filter: (base.audience_member_id =
email.audience_member_id)"
=================================================================================================
Questions here are :
Is the query written correctly as per the PostgreSQL?
Am I missing anything here?
Total Memory : 8 GB
shared_buffers = 2GB
work_mem = 64MB
maintenance_work_mem = 700MB
effective_cache_size = 4GB
Any kind of help is appreciated.
Warm Regards,
Vivekanand Joshi
+919654227927
[image: Zeta Interactive]
185 Madison Ave. New York, NY 10016
www.zetainteractive.com
Attachments:
image001.png (image/png, 184B) download
view image
* Re: Performance issues
2015-03-13 11:44 Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
@ 2015-03-13 12:58 ` Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 13:03 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
0 siblings, 1 reply; 59+ messages in thread
From: Varadharajan Mukundan @ 2015-03-13 12:58 UTC (permalink / raw)
To: vjoshi@zetainteractive.com; +Cc: pgsql-performance
Hi Vivekanand,
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Performance issues
2015-03-13 11:44 Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 12:58 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
@ 2015-03-13 13:03 ` Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 13:10 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
0 siblings, 1 reply; 59+ messages in thread
From: Vivekanand Joshi @ 2015-03-13 13:03 UTC (permalink / raw)
To: Varadharajan Mukundan <srinathsmn@gmail.com>; +Cc: pgsql-performance
10 million records in s_f_promotion_history table.
*From:* Varadharajan Mukundan [mailto:srinathsmn@gmail.com]
*Sent:* Friday, March 13, 2015 6:29 PM
*To:* vjoshi@zetainteractive.com
*Cc:* pgsql-performance@postgresql.org
*Subject:* Re: [PERFORM] Performance issues
Hi Vivekanand,
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Performance issues
2015-03-13 11:44 Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 12:58 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 13:03 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
@ 2015-03-13 13:10 ` Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 19:59 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
0 siblings, 1 reply; 59+ messages in thread
From: Varadharajan Mukundan @ 2015-03-13 13:10 UTC (permalink / raw)
To: vjoshi@zetainteractive.com; +Cc: pgsql-performance
If the s_f_promotion_history table will have a explosive growth, then its
worth considering partitioning by date and using constraint exclusion to
speed up the queries. Else, it makes sense to get started with multiple
partial index (like, have a index for each week or something like that. You
may want to start with a coarse grain timeline for the index and then fine
grain it based on the needs)
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Performance issues
2015-03-13 11:44 Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 12:58 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 13:03 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 13:10 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
@ 2015-03-13 19:59 ` Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 20:36 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 1 reply; 59+ messages in thread
From: Vivekanand Joshi @ 2015-03-13 19:59 UTC (permalink / raw)
To: Varadharajan Mukundan <srinathsmn@gmail.com>; +Cc: pgsql-performance
I am really worried about the performance of PostgreSQL as we have almost
1.5 billion records in promotion history table. Do you guys really think
PostgreSQL can handle this much load. We have fact tables which are more
than 15 GB in size and we have to make joins with those tables in almost
every query.
On 13 Mar 2015 18:40, "Varadharajan Mukundan" <srinathsmn@gmail.com> wrote:
> If the s_f_promotion_history table will have a explosive growth, then its
> worth considering partitioning by date and using constraint exclusion to
> speed up the queries. Else, it makes sense to get started with multiple
> partial index (like, have a index for each week or something like that. You
> may want to start with a coarse grain timeline for the index and then fine
> grain it based on the needs)
>
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Performance issues
2015-03-13 11:44 Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 12:58 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 13:03 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 13:10 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 19:59 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
@ 2015-03-13 20:36 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-13 20:46 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
0 siblings, 1 reply; 59+ messages in thread
From: Tomas Vondra @ 2015-03-13 20:36 UTC (permalink / raw)
To: pgsql-performance
Hi,
On 13.3.2015 20:59, Vivekanand Joshi wrote:
> I am really worried about the performance of PostgreSQL as we have
> almost 1.5 billion records in promotion history table. Do you guys
In the previous message you claimed the post table has 10M rows ...
> really think PostgreSQL can handle this much load. We have fact
> tables which are more than 15 GB in size and we have to make joins
> with those tables in almost every query.
That depends on what performance you're looking for. You'll have to
provide considerably more information until we can help you. You might
want to check this:
https://wiki.postgresql.org/wiki/Slow_Query_Questions
You have not provided the full query, just a query apparently
referencing views, so that the actual query is way more complicated.
Also, plain EXPLAIN is not really sufficient, we need EXPLAIN ANALYZE.
regards
--
Tomas Vondra http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Performance issues
2015-03-13 11:44 Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 12:58 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 13:03 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 13:10 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 19:59 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 20:36 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
@ 2015-03-13 20:46 ` Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 21:20 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-13 22:03 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
0 siblings, 2 replies; 59+ messages in thread
From: Vivekanand Joshi @ 2015-03-13 20:46 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: pgsql-performance
Since I was doing it only for the testing purposes and on a development
server which has only 8 GB of RAM, I used only 10m rows. But the original
table has 1.5 billion rows. We will obviously be using a server with very
high capacity, but I am not satisfied with the performance at all. This
might be only a start, so I might get a better performance later.
Yes, the view is complex and almost is created by using 10 tables. Same
goes with other views as well but this is what we are using in Netezza as
well. And we are getting results of the full report in less than 5 seconds.
And add to that, this is only a very little part of the whole query used in
a report.
I will post the result of whole query with Explain analyze tomorrow.
We might even consider taking experts advice on how to tune queries and
server, but if postgres is going to behave like this, I am not sure we
would be able to continue with it.
Having said that, I would day again that I am completely new to this
territory, so I might miss lots and lots of thing.
On 14 Mar 2015 02:07, "Tomas Vondra" <tomas.vondra@2ndquadrant.com> wrote:
> Hi,
>
> On 13.3.2015 20:59, Vivekanand Joshi wrote:
> > I am really worried about the performance of PostgreSQL as we have
> > almost 1.5 billion records in promotion history table. Do you guys
>
> In the previous message you claimed the post table has 10M rows ...
>
> > really think PostgreSQL can handle this much load. We have fact
> > tables which are more than 15 GB in size and we have to make joins
> > with those tables in almost every query.
>
> That depends on what performance you're looking for. You'll have to
> provide considerably more information until we can help you. You might
> want to check this:
>
> https://wiki.postgresql.org/wiki/Slow_Query_Questions
>
> You have not provided the full query, just a query apparently
> referencing views, so that the actual query is way more complicated.
> Also, plain EXPLAIN is not really sufficient, we need EXPLAIN ANALYZE.
>
> regards
>
> --
> Tomas Vondra http://www.2ndQuadrant.com/
> PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
>
>
> --
> Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-performance
>
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Performance issues
2015-03-13 11:44 Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 12:58 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 13:03 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 13:10 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 19:59 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 20:36 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-13 20:46 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
@ 2015-03-13 21:20 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
1 sibling, 0 replies; 59+ messages in thread
From: Tomas Vondra @ 2015-03-13 21:20 UTC (permalink / raw)
To: vjoshi@zetainteractive.com; +Cc: pgsql-performance
On 13.3.2015 21:46, Vivekanand Joshi wrote:
> Since I was doing it only for the testing purposes and on a
> development server which has only 8 GB of RAM, I used only 10m rows.
> But the original table has 1.5 billion rows. We will obviously be
> using a server with very high capacity, but I am not satisfied with
> the performance at all. This might be only a start, so I might get a
> better performance later.
OK, understood.
> Yes, the view is complex and almost is created by using 10 tables. Same
> goes with other views as well but this is what we are using in Netezza
> as well. And we are getting results of the full report in less than 5
> seconds. And add to that, this is only a very little part of the whole
> query used in a report.
Well, in the very first message you asked "Is the query written
correctly as per the PostgreSQL?" - how can we decide that when most of
the query is hidden in some unknown view?
> I will post the result of whole query with Explain analyze tomorrow.
Please also collect some information about the system using iostat,
vmstat and such, so that we know what is the bottleneck.
> We might even consider taking experts advice on how to tune queries
> and server, but if postgres is going to behave like this, I am not
> sure we would be able to continue with it.
That's probably a good idea.
--
Tomas Vondra http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Performance issues
2015-03-13 11:44 Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 12:58 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 13:03 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 13:10 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 19:59 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 20:36 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-13 20:46 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
@ 2015-03-13 22:03 ` Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 22:25 ` Re: Performance issues Scott Marlowe <scott.marlowe@gmail.com>
1 sibling, 1 reply; 59+ messages in thread
From: Varadharajan Mukundan @ 2015-03-13 22:03 UTC (permalink / raw)
To: vjoshi@zetainteractive.com; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; pgsql-performance
> We might even consider taking experts advice on how to tune queries and
> server, but if postgres is going to behave like this, I am not sure we would
> be able to continue with it.
>
> Having said that, I would day again that I am completely new to this
> territory, so I might miss lots and lots of thing.
My two cents: Postgres out of the box might not be a good choice for
data warehouse style queries, that is because it is optimized to run
thousands of small queries (OLTP style processing) and not one big
monolithic query. I've faced similar problems myself before and here
are few tricks i followed to get my elephant do real time adhoc
analysis on a table with ~45 columns and few billion rows in it.
1. Partition your table! use constraint exclusion to the fullest extent
2. Fire multiple small queries distributed over partitions and
aggregate them at the application layer. This is needed because, you
might to exploit all your cores to the fullest extent (Assuming that
you've enough memory for effective FS cache). If your dataset goes
beyond the capability of a single system, try something like Stado
(GridSQL)
3. Storing index on a RAM / faster disk disk (using tablespaces) and
using it properly makes the system blazing fast. CAUTION: This
requires some other infrastructure setup for backup and recovery
4. If you're accessing a small set of columns in a big table and if
you feel compressing the data helps a lot, give this FDW a try -
https://github.com/citusdata/cstore_fdw
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Performance issues
2015-03-13 11:44 Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 12:58 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 13:03 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 13:10 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 19:59 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 20:36 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-13 20:46 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 22:03 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
@ 2015-03-13 22:25 ` Scott Marlowe <scott.marlowe@gmail.com>
2015-03-13 23:28 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
0 siblings, 1 reply; 59+ messages in thread
From: Scott Marlowe @ 2015-03-13 22:25 UTC (permalink / raw)
To: Varadharajan Mukundan <srinathsmn@gmail.com>; +Cc: vjoshi@zetainteractive.com, Tomas Vondra <tomas.vondra@2ndquadrant.com>; pgsql-performance
On Fri, Mar 13, 2015 at 4:03 PM, Varadharajan Mukundan
<srinathsmn@gmail.com> wrote:
>> We might even consider taking experts advice on how to tune queries and
>> server, but if postgres is going to behave like this, I am not sure we would
>> be able to continue with it.
>>
>> Having said that, I would day again that I am completely new to this
>> territory, so I might miss lots and lots of thing.
>
> My two cents: Postgres out of the box might not be a good choice for
> data warehouse style queries, that is because it is optimized to run
> thousands of small queries (OLTP style processing) and not one big
> monolithic query. I've faced similar problems myself before and here
> are few tricks i followed to get my elephant do real time adhoc
> analysis on a table with ~45 columns and few billion rows in it.
>
> 1. Partition your table! use constraint exclusion to the fullest extent
> 2. Fire multiple small queries distributed over partitions and
> aggregate them at the application layer. This is needed because, you
> might to exploit all your cores to the fullest extent (Assuming that
> you've enough memory for effective FS cache). If your dataset goes
> beyond the capability of a single system, try something like Stado
> (GridSQL)
> 3. Storing index on a RAM / faster disk disk (using tablespaces) and
> using it properly makes the system blazing fast. CAUTION: This
> requires some other infrastructure setup for backup and recovery
> 4. If you're accessing a small set of columns in a big table and if
> you feel compressing the data helps a lot, give this FDW a try -
> https://github.com/citusdata/cstore_fdw
Agreed here. IF you're gonna run reporting queries against postgresql
you have to optimize for fast seq scan stuff. I.e. an IO subsystem
that can read a big table in hundreds of megabytes per second.
Gigabytes if you can get it. A lot of spinning drives on a fast RAID
card or good software raid can do this on the cheapish, since a lot of
times you don't need big drives if you have a lot. 24 cheap 1TB drives
that each can read at ~100 MB/s can gang up on the data and you can
read a 100GB in a few seconds. But you can't deny physics. If you need
to read a 2TB table it's going to take time.
If you're only running 1 or 2 queries at a time, you can crank up the
work_mem to something crazy like 1GB even on an 8GB machine. Stopping
sorts from spilling to disk, or at least giving queries a big
playground to work in can make a huge difference. If you're gonna give
big work_mem then definitely limit connections to a handful. If you
need a lot of persistent connections then use a pooler.
The single biggest mistake people make in setting up reporting servers
on postgresql is thinking that the same hardware that worked well for
transactional stuff (a handful of SSDs and lots of memory) might not
help when you're working with TB data sets. The hardware you need
isn't the same, and using that for a reporting server is gonna result
in sub-optimal performance.
--
To understand recursion, one must first understand recursion.
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Performance issues
2015-03-13 11:44 Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 12:58 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 13:03 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 13:10 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 19:59 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 20:36 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-13 20:46 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 22:03 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 22:25 ` Re: Performance issues Scott Marlowe <scott.marlowe@gmail.com>
@ 2015-03-13 23:28 ` Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-14 00:12 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 1 reply; 59+ messages in thread
From: Vivekanand Joshi @ 2015-03-13 23:28 UTC (permalink / raw)
To: Scott Marlowe <scott.marlowe@gmail.com>; Varadharajan Mukundan <srinathsmn@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; pgsql-performance
Hi Guys,
So here is the full information attached as well as in the link provided
below:
http://pgsql.privatepaste.com/41207bea45
I can provide new information as well.
Would like to see if queries of these type can actually run in postgres
server?
If yes, what would be the minimum requirements for hardware? We would like
to migrate our whole solution on PostgreSQL as we can spend on hardware as
much as we can but working on a proprietary appliance is becoming very
difficult for us.
Vivek
-----Original Message-----
From: Scott Marlowe [mailto:scott.marlowe@gmail.com]
Sent: Saturday, March 14, 2015 3:56 AM
To: Varadharajan Mukundan
Cc: vjoshi@zetainteractive.com; Tomas Vondra;
pgsql-performance@postgresql.org
Subject: Re: [PERFORM] Performance issues
On Fri, Mar 13, 2015 at 4:03 PM, Varadharajan Mukundan
<srinathsmn@gmail.com> wrote:
>> We might even consider taking experts advice on how to tune queries
>> and server, but if postgres is going to behave like this, I am not
>> sure we would be able to continue with it.
>>
>> Having said that, I would day again that I am completely new to this
>> territory, so I might miss lots and lots of thing.
>
> My two cents: Postgres out of the box might not be a good choice for
> data warehouse style queries, that is because it is optimized to run
> thousands of small queries (OLTP style processing) and not one big
> monolithic query. I've faced similar problems myself before and here
> are few tricks i followed to get my elephant do real time adhoc
> analysis on a table with ~45 columns and few billion rows in it.
>
> 1. Partition your table! use constraint exclusion to the fullest
> extent 2. Fire multiple small queries distributed over partitions and
> aggregate them at the application layer. This is needed because, you
> might to exploit all your cores to the fullest extent (Assuming that
> you've enough memory for effective FS cache). If your dataset goes
> beyond the capability of a single system, try something like Stado
> (GridSQL)
> 3. Storing index on a RAM / faster disk disk (using tablespaces) and
> using it properly makes the system blazing fast. CAUTION: This
> requires some other infrastructure setup for backup and recovery 4. If
> you're accessing a small set of columns in a big table and if you feel
> compressing the data helps a lot, give this FDW a try -
> https://github.com/citusdata/cstore_fdw
Agreed here. IF you're gonna run reporting queries against postgresql you
have to optimize for fast seq scan stuff. I.e. an IO subsystem that can read
a big table in hundreds of megabytes per second.
Gigabytes if you can get it. A lot of spinning drives on a fast RAID card or
good software raid can do this on the cheapish, since a lot of times you
don't need big drives if you have a lot. 24 cheap 1TB drives that each can
read at ~100 MB/s can gang up on the data and you can read a 100GB in a few
seconds. But you can't deny physics. If you need to read a 2TB table it's
going to take time.
If you're only running 1 or 2 queries at a time, you can crank up the
work_mem to something crazy like 1GB even on an 8GB machine. Stopping sorts
from spilling to disk, or at least giving queries a big playground to work
in can make a huge difference. If you're gonna give big work_mem then
definitely limit connections to a handful. If you need a lot of persistent
connections then use a pooler.
The single biggest mistake people make in setting up reporting servers on
postgresql is thinking that the same hardware that worked well for
transactional stuff (a handful of SSDs and lots of memory) might not help
when you're working with TB data sets. The hardware you need isn't the same,
and using that for a reporting server is gonna result in sub-optimal
performance.
--
To understand recursion, one must first understand recursion.
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
Attachments:
Full Information.txt (text/plain, 51.5K) download
Query Used in report:
=========================================================================================================================
SELECT
Z.SENT_IND AS IS_SENT,
COALESCE(X.EMAILS, 0) AS EMAILS,
COALESCE(X.PERCENT, 0) PERCENT
FROM
(
SELECT
CASE WHEN SENT_IND <= 1
THEN 1
WHEN SENT_IND <= 5
THEN 2
WHEN SENT_IND <= 10
THEN 3
WHEN SENT_IND <= 15
THEN 4
WHEN SENT_IND <= 20
THEN 5
WHEN SENT_IND <= 30
THEN 6
WHEN SENT_IND <= 50
THEN 7
WHEN SENT_IND <= 75
THEN 8
WHEN SENT_IND <= 100
THEN 9
ELSE 10
END AS SEND_RANK,
COUNT(DISTINCT TARGET_ID) AS EMAILS,
(COUNT(DISTINCT TARGET_ID)+0.0) / (
SELECT
COUNT(DISTINCT TARGET_ID)
FROM
S_V_F_PROMOTION_HISTORY_EMAIL PH
INNER JOIN S_V_D_CAMPAIGN_HIERARCHY CH
ON PH.TOUCHPOINT_EXECUTION_ID = CH.TOUCHPOINT_EXECUTION_ID
WHERE
1=1
AND SEND_DT >= '2014-03-13'
AND SEND_DT <= '2015-03-14'
-- AND ($X{IN,CH.CAMPAIGN_INITIATIVE,param_CAMPAIGN_INITIATIVE_2} OR $X{IN,'--ALL INITIATIVES--',param_CAMPAIGN_INITIATIVE_2})
) AS PERCENT
FROM
(
SELECT
TARGET_ID,
COUNT(PROMO_HIST_ID) AS SENT_IND
FROM
S_V_F_PROMOTION_HISTORY_EMAIL PH
INNER JOIN S_V_D_CAMPAIGN_HIERARCHY CH
ON PH.TOUCHPOINT_EXECUTION_ID = CH.TOUCHPOINT_EXECUTION_ID
WHERE
1=1
AND SEND_DT >= '2014-01-01'
AND SEND_DT <= '2015-03-14'
AND ($X{IN, CH.CAMPAIGN_INITIATIVE, param_CAMPAIGN_INITIATIVE_2}
OR $X{IN, '--ALL INITIATIVES--', param_CAMPAIGN_INITIATIVE_2})
GROUP BY
TARGET_ID
) A
GROUP BY
SEND_RANK
) X
RIGHT OUTER JOIN (
SELECT
1 AS SEND_RANK,
'1' AS SENT_IND
UNION
SELECT
2,
'2-5'
UNION
SELECT
3,
'6-10'
UNION
SELECT
4,
'11-15'
UNION
SELECT
5,
'16-20'
UNION
SELECT
6,
'21-30'
UNION
SELECT
7,
'31-50'
UNION
SELECT
8,
'51-75'
UNION
SELECT
9,
'76-100'
UNION
SELECT
10,
'101 OR MORE'
) Z
ON Z.SEND_RANK = X.SEND_RANK
ORDER BY
Z.SEND_RANK ASC ;
=========================================================================================================================
Views Used in the campaigns:
S_V_F_PROMOTION_HISTORY_EMAIL
S_V_D_CAMPAIGN_HIERARCHY
=========================================================================================================================
S_V_F_PROMOTION_HISTORY_EMAIL:
WITH valid_executions AS (
SELECT ste.touchpoint_execution_id
FROM s_v_d_successful_touchpoint_executions ste
JOIN s_d_touchpoint_execution tpe ON ste.touchpoint_execution_id = tpe.touchpoint_execution_id
JOIN s_d_touchpoint tp ON tpe.touchpoint_id = tp.touchpoint_id AND tp.channel_type_id = 1
)
SELECT base.promo_hist_id,
base.audience_member_id,
base.target_id,
base.touchpoint_execution_id,
base.contact_group_id,
base.content_version_execution_id,
base.sent_ind,
CASE
WHEN email.sbounce_ind IS NOT NULL THEN email.sbounce_ind::integer
WHEN 0 IS NOT NULL THEN 0
ELSE NULL::integer
END AS sbounce_ind,
CASE
WHEN email.hbounce_ind IS NOT NULL THEN email.hbounce_ind::integer
WHEN 0 IS NOT NULL THEN 0
ELSE NULL::integer
END AS hbounce_ind,
CASE
WHEN email.opened_ind IS NOT NULL THEN email.opened_ind::integer
WHEN 0 IS NOT NULL THEN 0
ELSE NULL::integer
END AS opened_ind,
CASE
WHEN email.clicked_ind IS NOT NULL THEN email.clicked_ind::integer
WHEN 0 IS NOT NULL THEN 0
ELSE NULL::integer
END AS clicked_ind,
CASE
WHEN email.unsubscribe_ind IS NOT NULL THEN email.unsubscribe_ind::integer
WHEN 0 IS NOT NULL THEN 0
ELSE NULL::integer
END AS unsubscribe_ind,
CASE
WHEN email.unsubscribe_spam_ind IS NOT NULL THEN email.unsubscribe_spam_ind::integer
WHEN 0 IS NOT NULL THEN 0
ELSE NULL::integer
END AS unsubscribe_spam_ind,
CASE
WHEN email.gross_open_cnt IS NOT NULL THEN email.gross_open_cnt::integer
WHEN 0 IS NOT NULL THEN 0
ELSE NULL::integer
END AS gross_open_cnt,
CASE
WHEN email.distinct_click_cnt IS NOT NULL THEN email.distinct_click_cnt::integer
WHEN 0 IS NOT NULL THEN 0
ELSE NULL::integer
END AS distinct_click_cnt,
CASE
WHEN email.gross_click_cnt IS NOT NULL THEN email.gross_click_cnt::integer
WHEN 0 IS NOT NULL THEN 0
ELSE NULL::integer
END AS gross_click_cnt,
base.send_dt,
email.first_click_dt,
email.first_open_dt,
email.unsubscribe_dt,
email.unsubscribe_spam_dt,
email.bounce_dt,
CASE
WHEN email.creation_dt IS NOT NULL THEN email.creation_dt
WHEN base.creation_dt IS NOT NULL THEN base.creation_dt
ELSE NULL::timestamp without time zone
END AS creation_dt,
CASE
WHEN email.modified_dt IS NOT NULL THEN email.modified_dt
WHEN base.modified_dt IS NOT NULL THEN base.modified_dt
ELSE NULL::timestamp without time zone
END AS modified_dt
FROM s_f_promotion_history base
LEFT JOIN s_f_promotion_history_email email ON base.promo_hist_id = email.promo_hist_id AND base.audience_member_id =
email.audience_member_id
JOIN valid_executions ON base.touchpoint_execution_id = valid_executions.touchpoint_execution_id;
S_V_D_CAMPAIGN_HIERARCHY :
WITH valid_executions AS (
SELECT s_f_touchpoint_execution_status_history.touchpoint_execution_id
FROM s_f_touchpoint_execution_status_history
JOIN ( SELECT s_f_touchpoint_execution_status_history_1.touchpoint_execution_id,
max(s_f_touchpoint_execution_status_history_1.creation_dt) AS last_status_change
FROM s_f_touchpoint_execution_status_history s_f_touchpoint_execution_status_history_1
GROUP BY s_f_touchpoint_execution_status_history_1.touchpoint_execution_id) tmp ON s_f_touchpoint_execut
ion_status_history.touchpoint_execution_id = tmp.touchpoint_execution_id AND s_f_touchpoint_execution_status_history.creat
ion_dt = tmp.last_status_change AND (s_f_touchpoint_execution_status_history.touchpoint_execution_status_type_id = ANY (AR
RAY[3, 4]))
)
SELECT camp.campaign_id,
camp.campaign_name,
camp.initiative AS campaign_initiative,
camp.objective AS campaign_objective,
camp.category_id AS campaign_category_id,
"CATEGORY".category_name AS campaign_category_name,
camp_exec.campaign_execution_id,
camp_exec.campaign_execution_name,
camp_exec.group_id,
grup.group_name,
camp_exec.start_dt AS campaign_execution_start_dt,
wave.wave_id,
wave.wave_name,
wave_exec.wave_execution_id,
wave_exec.wave_execution_name,
wave_exec.start_dt AS wave_execution_start_dt,
tp.touchpoint_id,
tp.touchpoint_name,
tp_exec.touchpoint_execution_id,
tp_exec.start_dt AS touchpoint_execution_start_dt,
channel.channel_type_id,
channel.channel_type_name,
tp_exec.message_type_id,
message_type.message_type_name,
tp_exec.content_id,
content.content_name
FROM s_d_touchpoint_execution tp_exec
JOIN s_d_wave_execution wave_exec ON tp_exec.wave_execution_id = wave_exec.wave_execution_id
JOIN s_d_campaign_execution camp_exec ON wave_exec.campaign_execution_id = camp_exec.campaign_execution_id
JOIN s_d_touchpoint tp ON tp_exec.touchpoint_id = tp.touchpoint_id
JOIN s_d_wave wave ON wave_exec.wave_id = wave.wave_id AND tp.wave_id = wave.wave_id
JOIN s_d_campaign camp ON camp_exec.campaign_id = camp.campaign_id AND wave.campaign_id = camp.campaign_id
LEFT JOIN s_d_content content ON tp_exec.content_id = content.content_id
LEFT JOIN s_d_message_type message_type ON tp_exec.message_type_id = message_type.message_type_id
LEFT JOIN s_d_group grup ON camp_exec.group_id = grup.group_id
JOIN s_d_channel_type channel ON tp.channel_type_id = channel.channel_type_id
LEFT JOIN s_d_category "CATEGORY" ON camp.category_id = "CATEGORY".category_id
JOIN valid_executions ON tp_exec.touchpoint_execution_id = valid_executions.touchpoint_execution_id;
=========================================================================================================================
Tables used in views along with the counts: These counts are for the testing server.
S_V_F_PROMOTION_HISTORY_EMAIL:
s_v_d_successful_touchpoint_executions (This is again a view). See definition below.
s_d_touchpoint_execution - 72956
s_d_touchpoint - 10972
s_f_promotion_history - 10000000
s_f_promotion_history_email - 46971
S_V_D_CAMPAIGN_HIERARCHY:
s_f_touchpoint_execution_status_history -- 291104
s_d_touchpoint_execution -- 72956
s_d_wave_execution -- 46247
s_d_campaign_execution -- 46244
s_d_touchpoint -- 10792
s_d_wave -- 9455
s_d_campaign -- 9433
s_d_content - 3180
s_d_message_type - 3
s_d_group - 7
s_d_channel_type - 5
s_d_category - 2
View Definition:
s_v_d_successful_touchpoint_executions:
SELECT DISTINCT s_f_touchpoint_execution_status_history.touchpoint_execution_id
FROM s_f_touchpoint_execution_status_history
JOIN ( SELECT s_f_touchpoint_execution_status_history_1.touchpoint_execution_id,
max(s_f_touchpoint_execution_status_history_1.creation_dt) AS last_status_change
FROM s_f_touchpoint_execution_status_history s_f_touchpoint_execution_status_history_1
GROUP BY s_f_touchpoint_execution_status_history_1.touchpoint_execution_id) tmp ON s_f_touchpoint_execution_status_history.touchpoint_execution_id = tmp.touchpoint_execution_id AND s_f_touchpoint_execution_status_history.creation_dt = tmp.last_status_change AND (s_f_touchpoint_execution_status_history.touchpoint_execution_status_type_id = ANY (ARRAY[3, 4, 6]));
Table Used:
s_f_touchpoint_execution_status_history
Count: 291104
=========================================================================================================================
Table definitions:
S_F_PROMOTION_HISTORY
TABLE "PUBLIC.S_F_PROMOTION_HISTORY"
COLUMN | TYPE | MODIFIERS | STORAGE | STATS TARGET | DESCRIPTION
------------------------------+-----------------------------+--------------------+---------+--------------+-------------
PROMO_HIST_ID | BIGINT | NOT NULL | PLAIN | |
TARGET_ID | BIGINT | NOT NULL | PLAIN | |
AUDIENCE_MEMBER_ID | BIGINT | NOT NULL | PLAIN | |
TOUCHPOINT_EXECUTION_ID | BIGINT | NOT NULL | PLAIN | |
CONTACT_GROUP_ID | BIGINT | NOT NULL | PLAIN | |
CONTENT_VERSION_EXECUTION_ID | BIGINT | NOT NULL | PLAIN | |
SENT_IND | SMALLINT | NOT NULL DEFAULT 0 | PLAIN | |
SEND_DT | TIMESTAMP WITHOUT TIME ZONE | | PLAIN | |
CREATION_DT | TIMESTAMP WITHOUT TIME ZONE | NOT NULL | PLAIN | |
MODIFIED_DT | TIMESTAMP WITHOUT TIME ZONE | NOT NULL | PLAIN | |
INDEXES:
"S_F_PROMOTION_HISTORY_IDX_PR" BTREE (PROMO_HIST_ID)
HAS OIDS: NO
S_F_PROMOTION_HISTORY_EMAIL
TABLE "PUBLIC.S_F_PROMOTION_HISTORY_EMAIL"
COLUMN | TYPE | MODIFIERS | STORAGE | STATS TARGET | DESCRIPTION
-------------------------+-----------------------------+--------------------+---------+--------------+-------------
PROMO_HIST_ID | BIGINT | NOT NULL | PLAIN | |
TARGET_ID | BIGINT | NOT NULL | PLAIN | |
AUDIENCE_MEMBER_ID | BIGINT | NOT NULL | PLAIN | |
TOUCHPOINT_EXECUTION_ID | BIGINT | NOT NULL | PLAIN | |
CONTACT_GROUP_ID | BIGINT | NOT NULL | PLAIN | |
SBOUNCE_IND | SMALLINT | NOT NULL DEFAULT 0 | PLAIN | |
HBOUNCE_IND | SMALLINT | NOT NULL DEFAULT 0 | PLAIN | |
OPENED_IND | SMALLINT | NOT NULL DEFAULT 0 | PLAIN | |
CLICKED_IND | SMALLINT | NOT NULL DEFAULT 0 | PLAIN | |
UNSUBSCRIBE_IND | SMALLINT | NOT NULL DEFAULT 0 | PLAIN | |
UNSUBSCRIBE_SPAM_IND | SMALLINT | NOT NULL DEFAULT 0 | PLAIN | |
GROSS_OPEN_CNT | SMALLINT | NOT NULL DEFAULT 0 | PLAIN | |
DISTINCT_CLICK_CNT | SMALLINT | NOT NULL DEFAULT 0 | PLAIN | |
GROSS_CLICK_CNT | SMALLINT | NOT NULL DEFAULT 0 | PLAIN | |
FIRST_CLICK_DT | TIMESTAMP WITHOUT TIME ZONE | | PLAIN | |
FIRST_OPEN_DT | TIMESTAMP WITHOUT TIME ZONE | | PLAIN | |
UNSUBSCRIBE_DT | TIMESTAMP WITHOUT TIME ZONE | | PLAIN | |
UNSUBSCRIBE_SPAM_DT | TIMESTAMP WITHOUT TIME ZONE | | PLAIN | |
BOUNCE_DT | TIMESTAMP WITHOUT TIME ZONE | | PLAIN | |
CREATION_DT | TIMESTAMP WITHOUT TIME ZONE | NOT NULL | PLAIN | |
MODIFIED_DT | TIMESTAMP WITHOUT TIME ZONE | NOT NULL | PLAIN | |
INDEXES:
"S_F_PROMOTION_HISTORY_EMAIL_PK1" PRIMARY KEY, BTREE (PROMO_HIST_ID)
HAS OIDS: NO
S_F_TOUCHPOINT_EXECUTION_STATUS_HISTORY
TABLE "PUBLIC.S_F_TOUCHPOINT_EXECUTION_STATUS_HISTORY"
COLUMN | TYPE | MODIFIERS | STORAGE | STATS TARGET | DESCRIPTION
-------------------------------------+-----------------------------+----------------------------------------+----------+--------------+-------------
TOUCHPOINT_EXECUTION_ID | BIGINT | NOT NULL | PLAIN | |
TOUCHPOINT_EXECUTION_STATUS_TYPE_ID | INTEGER | NOT NULL | PLAIN | |
STATUS_MESSAGE | CHARACTER VARYING(255) | NOT NULL DEFAULT ''::CHARACTER VARYING | EXTENDED | |
CREATION_DT | TIMESTAMP WITHOUT TIME ZONE | NOT NULL | PLAIN | |
HAS OIDS: NO
S_D_TOUCHPOINT_EXECUTION
TABLE "PUBLIC.S_D_TOUCHPOINT_EXECUTION"
COLUMN | TYPE | MODIFIERS
-------------------------+-----------------------------+-----------
TOUCHPOINT_EXECUTION_ID | BIGINT | NOT NULL
WAVE_EXECUTION_ID | BIGINT | NOT NULL
TOUCHPOINT_ID | BIGINT | NOT NULL
CHANNEL_TYPE_ID | SMALLINT | NOT NULL
CONTENT_ID | BIGINT | NOT NULL
MESSAGE_TYPE_ID | SMALLINT | NOT NULL
START_DT | TIMESTAMP WITHOUT TIME ZONE | NOT NULL
CREATION_DT | TIMESTAMP WITHOUT TIME ZONE | NOT NULL
INDEXES:
"S_D_TOUCHPOINT_EXECUTION_PKEY" PRIMARY KEY, BTREE (TOUCHPOINT_EXECUTION_ID, CHANNEL_TYPE_ID)
S_D_WAVE_EXECUTION
TABLE "PUBLIC.S_D_WAVE_EXECUTION"
COLUMN | TYPE | MODIFIERS | STORAGE | STATS TARGET | DESCRIPTION
-----------------------+-----------------------------+-----------+----------+--------------+-------------
WAVE_EXECUTION_ID | BIGINT | NOT NULL | PLAIN | |
WAVE_ID | BIGINT | NOT NULL | PLAIN | |
CAMPAIGN_EXECUTION_ID | BIGINT | NOT NULL | PLAIN | |
WAVE_EXECUTION_NAME | CHARACTER VARYING(300) | NOT NULL | EXTENDED | |
START_DT | TIMESTAMP WITHOUT TIME ZONE | NOT NULL | PLAIN | |
CREATION_DT | TIMESTAMP WITHOUT TIME ZONE | NOT NULL | PLAIN | |
INDEXES:
"WAVE_EXECUTION_ID_IDX" UNIQUE, BTREE (WAVE_EXECUTION_ID)
HAS OIDS: NO
S_D_CAMPAIGN_EXECUTION
TABLE "PUBLIC.S_D_CAMPAIGN_EXECUTION"
COLUMN | TYPE | MODIFIERS | STORAGE | STATS TARGET | DESCRIPTION
-------------------------+-----------------------------+-----------+----------+--------------+-------------
CAMPAIGN_EXECUTION_ID | BIGINT | NOT NULL | PLAIN | |
CAMPAIGN_ID | BIGINT | NOT NULL | PLAIN | |
GROUP_ID | SMALLINT | NOT NULL | PLAIN | |
CAMPAIGN_EXECUTION_NAME | CHARACTER VARYING(300) | NOT NULL | EXTENDED | |
START_DT | TIMESTAMP WITHOUT TIME ZONE | NOT NULL | PLAIN | |
CREATION_DT | TIMESTAMP WITHOUT TIME ZONE | NOT NULL | PLAIN | |
INDEXES:
"S_D_CAMPAIGN_EXECUTION_IDX" BTREE (CAMPAIGN_EXECUTION_ID)
HAS OIDS: NO
S_D_TOUCHPOINT
TABLE "PUBLIC.S_D_TOUCHPOINT"
COLUMN | TYPE | MODIFIERS | STORAGE | STATS TARGET | DESCRIPTION
-----------------+-----------------------------+-----------+----------+--------------+-------------
TOUCHPOINT_ID | BIGINT | NOT NULL | PLAIN | |
WAVE_ID | BIGINT | NOT NULL | PLAIN | |
CHANNEL_TYPE_ID | SMALLINT | NOT NULL | PLAIN | |
TOUCHPOINT_NAME | CHARACTER VARYING(255) | NOT NULL | EXTENDED | |
CREATION_DT | TIMESTAMP WITHOUT TIME ZONE | NOT NULL | PLAIN | |
MODIFIED_DT | TIMESTAMP WITHOUT TIME ZONE | NOT NULL | PLAIN | |
INDEXES:
"S_D_TOUCHPOINT_PKEY" PRIMARY KEY, BTREE (TOUCHPOINT_ID, CHANNEL_TYPE_ID)
HAS OIDS: NO
S_D_WAVE
TABLE "PUBLIC.S_D_WAVE"
COLUMN | TYPE | MODIFIERS | STORAGE | STATS TARGET | DESCRIPTION
-------------+-----------------------------+-----------+----------+--------------+-------------
WAVE_ID | BIGINT | NOT NULL | PLAIN | |
CAMPAIGN_ID | BIGINT | NOT NULL | PLAIN | |
WAVE_NAME | CHARACTER VARYING(255) | NOT NULL | EXTENDED | |
CREATION_DT | TIMESTAMP WITHOUT TIME ZONE | NOT NULL | PLAIN | |
MODIFIED_DT | TIMESTAMP WITHOUT TIME ZONE | NOT NULL | PLAIN | |
INDEXES:
"S_D_WAVE_PKEY" PRIMARY KEY, BTREE (WAVE_ID)
HAS OIDS: NO
S_D_CAMPAIGN
TABLE "PUBLIC.S_D_CAMPAIGN"
COLUMN | TYPE | MODIFIERS | STORAGE | STATS TARGET | DESCRIPTION
---------------+-----------------------------+-----------+----------+--------------+-------------
CAMPAIGN_ID | BIGINT | NOT NULL | PLAIN | |
CAMPAIGN_NAME | CHARACTER VARYING(150) | NOT NULL | EXTENDED | |
OBJECTIVE | CHARACTER VARYING(256) | | EXTENDED | |
INITIATIVE | CHARACTER VARYING(256) | | EXTENDED | |
CATEGORY_ID | SMALLINT | NOT NULL | PLAIN | |
CREATION_DT | TIMESTAMP WITHOUT TIME ZONE | NOT NULL | PLAIN | |
MODIFIED_DT | TIMESTAMP WITHOUT TIME ZONE | NOT NULL | PLAIN | |
INDEXES:
"S_D_CAMPAIGN_PKEY" PRIMARY KEY, BTREE (CAMPAIGN_ID)
HAS OIDS: NO
S_D_CONTENT
TABLE "PUBLIC.S_D_CONTENT"
COLUMN | TYPE | MODIFIERS | STORAGE | STATS TARGET | DESCRIPTION
-----------------+-----------------------------+-----------+----------+--------------+-------------
CONTENT_ID | BIGINT | NOT NULL | PLAIN | |
CONTENT_NAME | CHARACTER VARYING(255) | NOT NULL | EXTENDED | |
CREATION_DT | TIMESTAMP WITHOUT TIME ZONE | NOT NULL | PLAIN | |
CHANNEL_TYPE_ID | SMALLINT | NOT NULL | PLAIN | |
MODIFIED_DT | TIMESTAMP WITHOUT TIME ZONE | NOT NULL | PLAIN | |
INDEXES:
"S_D_CONTENT_PKEY" PRIMARY KEY, BTREE (CONTENT_ID, CHANNEL_TYPE_ID)
HAS OIDS: NO
S_D_MESSAGE_TYPE
TABLE "PUBLIC.S_D_MESSAGE_TYPE"
COLUMN | TYPE | MODIFIERS | STORAGE | STATS TARGET | DESCRIPTION
-------------------+-----------------------------+-----------+----------+--------------+-------------
MESSAGE_TYPE_ID | SMALLINT | NOT NULL | PLAIN | |
MESSAGE_TYPE_NAME | CHARACTER VARYING(50) | NOT NULL | EXTENDED | |
CREATION_DT | TIMESTAMP WITHOUT TIME ZONE | NOT NULL | PLAIN | |
MODIFIED_DT | TIMESTAMP WITHOUT TIME ZONE | NOT NULL | PLAIN | |
INDEXES:
"S_D_MESSAGE_TYPE_PKEY" PRIMARY KEY, BTREE (MESSAGE_TYPE_ID)
HAS OIDS: NO
S_D_GROUP
TABLE "PUBLIC.S_D_GROUP"
COLUMN | TYPE | MODIFIERS | STORAGE | STATS TARGET | DESCRIPTION
-------------+-----------------------------+-----------+----------+--------------+-------------
GROUP_ID | SMALLINT | NOT NULL | PLAIN | |
GROUP_NAME | CHARACTER VARYING(150) | NOT NULL | EXTENDED | |
CREATION_DT | TIMESTAMP WITHOUT TIME ZONE | NOT NULL | PLAIN | |
MODIFIED_DT | TIMESTAMP WITHOUT TIME ZONE | NOT NULL | PLAIN | |
INDEXES:
"S_D_GROUP_PKEY" PRIMARY KEY, BTREE (GROUP_ID)
HAS OIDS: NO
S_D_CHANNEL_TYPE
COLUMN | TYPE | MODIFIERS | STORAGE | STATS TARGET | DESCRIPTION
-------------------+-----------------------+-----------+----------+--------------+-------------
CHANNEL_TYPE_ID | SMALLINT | NOT NULL | PLAIN | |
CHANNEL_TYPE_NAME | CHARACTER VARYING(50) | NOT NULL | EXTENDED | |
INDEXES:
"D_CHANNEL_PK" PRIMARY KEY, BTREE (CHANNEL_TYPE_ID)
HAS OIDS: NO
S_D_CATEGORY
TABLE "PUBLIC.S_D_CATEGORY"
COLUMN | TYPE | MODIFIERS | STORAGE | STATS TARGET | DESCRIPTION
---------------+-----------------------------+-----------+----------+--------------+-------------
CATEGORY_ID | SMALLINT | NOT NULL | PLAIN | |
CATEGORY_NAME | CHARACTER VARYING(50) | NOT NULL | EXTENDED | |
CREATION_DT | TIMESTAMP WITHOUT TIME ZONE | NOT NULL | PLAIN | |
MODIFIED_DT | TIMESTAMP WITHOUT TIME ZONE | NOT NULL | PLAIN | |
INDEXES:
"S_D_CATEGORY_PKEY" PRIMARY KEY, BTREE (CATEGORY_ID)
HAS OIDS: NO
=========================================================================================================================
Explain for the Query:
"Merge Left Join (cost=711668.51..711668.64 rows=10 width=76)"
" Merge Cond: (z.send_rank = (CASE WHEN (a.sent_ind <= 1) THEN 1 WHEN (a.sent_ind <= 5) THEN 2 WHEN (a.sent_ind <= 10) THEN 3 WHEN (a.sent_ind <= 15) THEN 4 WHEN (a.sent_ind <= 20) THEN 5 WHEN (a.sent_ind <= 30) THEN 6 WHEN (a.sent_ind <= 50) THEN 7 WHEN ( (...)"
" -> Sort (cost=0.62..0.64 rows=10 width=36)"
" Sort Key: z.send_rank"
" -> Subquery Scan on z (cost=0.25..0.45 rows=10 width=36)"
" -> HashAggregate (cost=0.25..0.35 rows=10 width=0)"
" -> Append (cost=0.00..0.20 rows=10 width=0)"
" -> Result (cost=0.00..0.01 rows=1 width=0)"
" -> Result (cost=0.00..0.01 rows=1 width=0)"
" -> Result (cost=0.00..0.01 rows=1 width=0)"
" -> Result (cost=0.00..0.01 rows=1 width=0)"
" -> Result (cost=0.00..0.01 rows=1 width=0)"
" -> Result (cost=0.00..0.01 rows=1 width=0)"
" -> Result (cost=0.00..0.01 rows=1 width=0)"
" -> Result (cost=0.00..0.01 rows=1 width=0)"
" -> Result (cost=0.00..0.01 rows=1 width=0)"
" -> Result (cost=0.00..0.01 rows=1 width=0)"
" -> Materialize (cost=711667.89..711667.96 rows=1 width=44)"
" -> GroupAggregate (cost=711667.89..711667.95 rows=1 width=16)"
" InitPlan 3 (returns $14)"
" -> Aggregate (cost=355832.51..355832.52 rows=1 width=8)"
" -> Nested Loop (cost=68171.53..355832.51 rows=1 width=8)"
" Join Filter: (base_1.touchpoint_execution_id = tp_exec_1.touchpoint_execution_id)"
" -> Nested Loop (cost=33666.96..37971.39 rows=1 width=894)"
" Join Filter: (tp_exec_1.touchpoint_execution_id = valid_executions_2.touchpoint_execution_id)"
" CTE valid_executions"
" -> Hash Join (cost=13753.53..31711.17 rows=1 width=8)"
" Hash Cond: ((s_f_touchpoint_execution_status_history_1_1.touchpoint_execution_id = s_f_touchpoint_execution_status_history_2.touchpoint_execution_id) AND ((max(s_f_touchpoint_execution_status_history_1_1.creation_d (...)"
" -> HashAggregate (cost=6221.56..6905.66 rows=68410 width=16)"
" -> Seq Scan on s_f_touchpoint_execution_status_history s_f_touchpoint_execution_status_history_1_1 (cost=0.00..4766.04 rows=291104 width=16)"
" -> Hash (cost=5493.80..5493.80 rows=135878 width=16)"
" -> Seq Scan on s_f_touchpoint_execution_status_history s_f_touchpoint_execution_status_history_2 (cost=0.00..5493.80 rows=135878 width=16)"
" Filter: (touchpoint_execution_status_type_id = ANY ('{3,4}'::integer[]))"
" -> Nested Loop Left Join (cost=1955.80..6260.19 rows=1 width=894)"
" -> Nested Loop (cost=1955.67..6260.04 rows=1 width=776)"
" -> Nested Loop Left Join (cost=1955.54..6259.87 rows=1 width=658)"
" -> Nested Loop Left Join (cost=1955.40..6259.71 rows=1 width=340)"
" -> Nested Loop Left Join (cost=1955.27..6259.55 rows=1 width=222)"
" -> Nested Loop (cost=1954.99..6259.24 rows=1 width=197)"
" -> Nested Loop (cost=1954.71..6258.92 rows=1 width=173)"
" Join Filter: (camp_exec_1.campaign_id = wave_1.campaign_id)"
" -> Nested Loop (cost=1954.42..6254.67 rows=13 width=167)"
" -> Hash Join (cost=1954.13..6249.67 rows=13 width=108)"
" Hash Cond: ((tp_exec_1.touchpoint_id = tp_2.touchpoint_id) AND (wave_exec_1.wave_id = tp_2.wave_id))"
" -> Hash Join (cost=1576.83..4595.51 rows=72956 width=90)"
" Hash Cond: (tp_exec_1.wave_execution_id = wave_exec_1.wave_execution_id)"
" -> Seq Scan on s_d_touchpoint_execution tp_exec_1 (cost=0.00..1559.56 rows=72956 width=42)"
" -> Hash (cost=1001.37..1001.37 rows=46037 width=56)"
" -> Seq Scan on s_d_wave_execution wave_exec_1 (cost=0.00..1001.37 rows=46037 width=56)"
" -> Hash (cost=212.72..212.72 rows=10972 width=26)"
" -> Seq Scan on s_d_touchpoint tp_2 (cost=0.00..212.72 rows=10972 width=26)"
" -> Index Scan using s_d_campaign_execution_idx on s_d_campaign_execution camp_exec_1 (cost=0.29..0.37 rows=1 width=67)"
" Index Cond: (campaign_execution_id = wave_exec_1.campaign_execution_id)"
" -> Index Scan using s_d_wave_pkey on s_d_wave wave_1 (cost=0.29..0.31 rows=1 width=22)"
" Index Cond: (wave_id = wave_exec_1.wave_id)"
" -> Index Scan using s_d_campaign_pkey on s_d_campaign camp_1 (cost=0.29..0.32 rows=1 width=40)"
" Index Cond: (campaign_id = camp_exec_1.campaign_id)"
" -> Index Scan using s_d_content_pkey on s_d_content content_1 (cost=0.28..0.30 rows=1 width=33)"
" Index Cond: (tp_exec_1.content_id = content_id)"
" -> Index Scan using s_d_message_type_pkey on s_d_message_type message_type_1 (cost=0.13..0.15 rows=1 width=120)"
" Index Cond: (tp_exec_1.message_type_id = message_type_id)"
" -> Index Scan using s_d_group_pkey on s_d_group grup_1 (cost=0.13..0.15 rows=1 width=320)"
" Index Cond: (camp_exec_1.group_id = group_id)"
" -> Index Scan using d_channel_pk on s_d_channel_type channel_1 (cost=0.13..0.15 rows=1 width=120)"
" Index Cond: (channel_type_id = tp_2.channel_type_id)"
" -> Index Scan using s_d_category_pkey on s_d_category "CATEGORY_1" (cost=0.13..0.15 rows=1 width=120)"
" Index Cond: (camp_1.category_id = category_id)"
" -> CTE Scan on valid_executions valid_executions_2 (cost=0.00..0.02 rows=1 width=8)"
" -> Nested Loop Left Join (cost=34504.57..317861.06 rows=2 width=148)"
" CTE valid_executions"
" -> Nested Loop (cost=32782.31..34504.28 rows=1 width=8)"
" -> Nested Loop (cost=32782.03..34503.96 rows=1 width=16)"
" -> Unique (cost=32781.61..34495.50 rows=1 width=8)"
" -> Merge Join (cost=32781.61..34495.50 rows=1 width=8)"
" Merge Cond: ((s_f_touchpoint_execution_status_history.touchpoint_execution_id = s_f_touchpoint_execution_status_history_1.touchpoint_execution_id) AND (s_f_touchpoint_execution_status_history.crea (...)"
" -> Sort (cost=19697.87..20098.14 rows=160107 width=16)"
" Sort Key: s_f_touchpoint_execution_status_history.touchpoint_execution_id, s_f_touchpoint_execution_status_history.creation_dt"
" -> Seq Scan on s_f_touchpoint_execution_status_history (cost=0.00..5857.68 rows=160107 width=16)"
" Filter: (touchpoint_execution_status_type_id = ANY ('{3,4,6}'::integer[]))"
" -> Sort (cost=13083.74..13254.76 rows=68410 width=16)"
" Sort Key: s_f_touchpoint_execution_status_history_1.touchpoint_execution_id, (max(s_f_touchpoint_execution_status_history_1.creation_dt))"
" -> HashAggregate (cost=6221.56..6905.66 rows=68410 width=16)"
" -> Seq Scan on s_f_touchpoint_execution_status_history s_f_touchpoint_execution_status_history_1 (cost=0.00..4766.04 rows=291104 width=16)"
" -> Index Scan using s_d_touchpoint_execution_pkey on s_d_touchpoint_execution tpe (cost=0.42..8.44 rows=1 width=16)"
" Index Cond: (touchpoint_execution_id = s_f_touchpoint_execution_status_history.touchpoint_execution_id)"
" -> Index Only Scan using s_d_touchpoint_pkey on s_d_touchpoint tp_1 (cost=0.29..0.32 rows=1 width=8)"
" Index Cond: ((touchpoint_id = tpe.touchpoint_id) AND (channel_type_id = 1))"
" -> Nested Loop (cost=0.00..283351.20 rows=2 width=74)"
" Join Filter: (base_1.touchpoint_execution_id = valid_executions_3.touchpoint_execution_id)"
" -> CTE Scan on valid_executions valid_executions_3 (cost=0.00..0.02 rows=1 width=8)"
" -> Seq Scan on s_f_promotion_history base_1 (cost=0.00..283334.17 rows=1361 width=74)"
" Filter: ((send_dt >= '2014-03-13 00:00:00'::timestamp without time zone) AND (send_dt <= '2015-03-14 00:00:00'::timestamp without time zone))"
" -> Index Scan using s_f_promotion_history_email_pk1 on s_f_promotion_history_email email_1 (cost=0.29..2.76 rows=1 width=90)"
" Index Cond: (base_1.promo_hist_id = promo_hist_id)"
" Filter: (base_1.audience_member_id = audience_member_id)"
" -> Sort (cost=355835.37..355835.38 rows=1 width=16)"
" Sort Key: (CASE WHEN (a.sent_ind <= 1) THEN 1 WHEN (a.sent_ind <= 5) THEN 2 WHEN (a.sent_ind <= 10) THEN 3 WHEN (a.sent_ind <= 15) THEN 4 WHEN (a.sent_ind <= 20) THEN 5 WHEN (a.sent_ind <= 30) THEN 6 WHEN (a.sent_ind <= 50) THEN 7 WHEN (...)"
" -> Subquery Scan on a (cost=355835.32..355835.36 rows=1 width=16)"
" -> HashAggregate (cost=355835.32..355835.33 rows=1 width=16)"
" -> Nested Loop (cost=68171.53..355835.31 rows=1 width=16)"
" Join Filter: (base.touchpoint_execution_id = tp_exec.touchpoint_execution_id)"
" -> Nested Loop (cost=33666.96..37971.39 rows=1 width=894)"
" Join Filter: (tp_exec.touchpoint_execution_id = valid_executions.touchpoint_execution_id)"
" CTE valid_executions"
" -> Hash Join (cost=13753.53..31711.17 rows=1 width=8)"
" Hash Cond: ((s_f_touchpoint_execution_status_history_1_3.touchpoint_execution_id = s_f_touchpoint_execution_status_history_4.touchpoint_execution_id) AND ((max(s_f_touchpoint_execution_status_history_1_3. (...)"
" -> HashAggregate (cost=6221.56..6905.66 rows=68410 width=16)"
" -> Seq Scan on s_f_touchpoint_execution_status_history s_f_touchpoint_execution_status_history_1_3 (cost=0.00..4766.04 rows=291104 width=16)"
" -> Hash (cost=5493.80..5493.80 rows=135878 width=16)"
" -> Seq Scan on s_f_touchpoint_execution_status_history s_f_touchpoint_execution_status_history_4 (cost=0.00..5493.80 rows=135878 width=16)"
" Filter: (touchpoint_execution_status_type_id = ANY ('{3,4}'::integer[]))"
" -> Nested Loop Left Join (cost=1955.80..6260.19 rows=1 width=894)"
" -> Nested Loop (cost=1955.67..6260.04 rows=1 width=776)"
" -> Nested Loop Left Join (cost=1955.54..6259.87 rows=1 width=658)"
" -> Nested Loop Left Join (cost=1955.40..6259.71 rows=1 width=340)"
" -> Nested Loop Left Join (cost=1955.27..6259.55 rows=1 width=222)"
" -> Nested Loop (cost=1954.99..6259.24 rows=1 width=197)"
" -> Nested Loop (cost=1954.71..6258.92 rows=1 width=173)"
" Join Filter: (camp_exec.campaign_id = wave.campaign_id)"
" -> Nested Loop (cost=1954.42..6254.67 rows=13 width=167)"
" -> Hash Join (cost=1954.13..6249.67 rows=13 width=108)"
" Hash Cond: ((tp_exec.touchpoint_id = tp.touchpoint_id) AND (wave_exec.wave_id = tp.wave_id))"
" -> Hash Join (cost=1576.83..4595.51 rows=72956 width=90)"
" Hash Cond: (tp_exec.wave_execution_id = wave_exec.wave_execution_id)"
" -> Seq Scan on s_d_touchpoint_execution tp_exec (cost=0.00..1559.56 rows=72956 width=42)"
" -> Hash (cost=1001.37..1001.37 rows=46037 width=56)"
" -> Seq Scan on s_d_wave_execution wave_exec (cost=0.00..1001.37 rows=46037 width=56)"
" -> Hash (cost=212.72..212.72 rows=10972 width=26)"
" -> Seq Scan on s_d_touchpoint tp (cost=0.00..212.72 rows=10972 width=26)"
" -> Index Scan using s_d_campaign_execution_idx on s_d_campaign_execution camp_exec (cost=0.29..0.37 rows=1 width=67)"
" Index Cond: (campaign_execution_id = wave_exec.campaign_execution_id)"
" -> Index Scan using s_d_wave_pkey on s_d_wave wave (cost=0.29..0.31 rows=1 width=22)"
" Index Cond: (wave_id = wave_exec.wave_id)"
" -> Index Scan using s_d_campaign_pkey on s_d_campaign camp (cost=0.29..0.32 rows=1 width=40)"
" Index Cond: (campaign_id = camp_exec.campaign_id)"
" -> Index Scan using s_d_content_pkey on s_d_content content (cost=0.28..0.30 rows=1 width=33)"
" Index Cond: (tp_exec.content_id = content_id)"
" -> Index Scan using s_d_message_type_pkey on s_d_message_type message_type (cost=0.13..0.15 rows=1 width=120)"
" Index Cond: (tp_exec.message_type_id = message_type_id)"
" -> Index Scan using s_d_group_pkey on s_d_group grup (cost=0.13..0.15 rows=1 width=320)"
" Index Cond: (camp_exec.group_id = group_id)"
" -> Index Scan using d_channel_pk on s_d_channel_type channel (cost=0.13..0.15 rows=1 width=120)"
" Index Cond: (channel_type_id = tp.channel_type_id)"
" -> Index Scan using s_d_category_pkey on s_d_category "CATEGORY" (cost=0.13..0.15 rows=1 width=120)"
" Index Cond: (camp.category_id = category_id)"
" -> CTE Scan on valid_executions (cost=0.00..0.02 rows=1 width=8)"
" -> Nested Loop Left Join (cost=34504.57..317863.87 rows=2 width=148)"
" CTE valid_executions"
" -> Nested Loop (cost=32782.31..34504.28 rows=1 width=8)"
" -> Nested Loop (cost=32782.03..34503.96 rows=1 width=16)"
" -> Unique (cost=32781.61..34495.50 rows=1 width=8)"
" -> Merge Join (cost=32781.61..34495.50 rows=1 width=8)"
" Merge Cond: ((s_f_touchpoint_execution_status_history_3.touchpoint_execution_id = s_f_touchpoint_execution_status_history_1_2.touchpoint_execution_id) AND (s_f_touchpoint_execution_statu (...)"
" -> Sort (cost=19697.87..20098.14 rows=160107 width=16)"
" Sort Key: s_f_touchpoint_execution_status_history_3.touchpoint_execution_id, s_f_touchpoint_execution_status_history_3.creation_dt"
" -> Seq Scan on s_f_touchpoint_execution_status_history s_f_touchpoint_execution_status_history_3 (cost=0.00..5857.68 rows=160107 width=16)"
" Filter: (touchpoint_execution_status_type_id = ANY ('{3,4,6}'::integer[]))"
" -> Sort (cost=13083.74..13254.76 rows=68410 width=16)"
" Sort Key: s_f_touchpoint_execution_status_history_1_2.touchpoint_execution_id, (max(s_f_touchpoint_execution_status_history_1_2.creation_dt))"
" -> HashAggregate (cost=6221.56..6905.66 rows=68410 width=16)"
" -> Seq Scan on s_f_touchpoint_execution_status_history s_f_touchpoint_execution_status_history_1_2 (cost=0.00..4766.04 rows=291104 width=16)"
" -> Index Scan using s_d_touchpoint_execution_pkey on s_d_touchpoint_execution tpe_1 (cost=0.42..8.44 rows=1 width=16)"
" Index Cond: (touchpoint_execution_id = s_f_touchpoint_execution_status_history_3.touchpoint_execution_id)"
" -> Index Only Scan using s_d_touchpoint_pkey on s_d_touchpoint tp_3 (cost=0.29..0.32 rows=1 width=8)"
" Index Cond: ((touchpoint_id = tpe_1.touchpoint_id) AND (channel_type_id = 1))"
" -> Nested Loop (cost=0.00..283354.75 rows=2 width=74)"
" Join Filter: (base.touchpoint_execution_id = valid_executions_1.touchpoint_execution_id)"
" -> CTE Scan on valid_executions valid_executions_1 (cost=0.00..0.02 rows=1 width=8)"
" -> Seq Scan on s_f_promotion_history base (cost=0.00..283334.17 rows=1645 width=74)"
" Filter: ((send_dt >= '2014-01-01 00:00:00'::timestamp without time zone) AND (send_dt <= '2015-03-14 00:00:00'::timestamp without time zone))"
" -> Index Scan using s_f_promotion_history_email_pk1 on s_f_promotion_history_email email (cost=0.29..2.39 rows=1 width=90)"
" Index Cond: (base.promo_hist_id = promo_hist_id)"
" Filter: (base.audience_member_id = audience_member_id)"
=========================================================================================================================
EXPLAIN ANALYZE: Query Never completed
Postgres version : PostgreSQL 9.3.5 on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 4.1.2 20080704 (Red Hat 4.1.2-54), 64-bit
History : Running for the first time.
Maintenance Setup: New Server and data has been just inserted.
WAL Configuration: Default Settings
shared_buffers = 6GB
work_mem = 1GB
maintenance_work_mem = 3GB
effective_cache_size = 16GB
All are commented:
#enable_bitmapscan = on
#enable_hashagg = on
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
#enable_seqscan = on
#enable_sort = on
#enable_tidscan = on
vmstat
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 0 548472 8193880 46160 5597832 0 0 8 34 0 0 1 0 99 0 0
==============
iostat
Linux 2.6.18-348.4.1.el5 (AT-PU-MERCURY) 03/13/2015
avg-cpu: %user %nice %system %iowait %steal %idle
0.54 0.00 0.15 0.04 0.00 99.26
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 5.72 62.89 262.98 3661856221 15313306008
sda1 0.00 0.02 0.00 911167 35000
sda2 5.72 62.87 262.98 3660943046 15313271008
dm-0 7.43 4.66 57.98 271216554 3375965736
dm-1 24.10 57.78 188.70 3364343202 10988093896
dm-2 1.72 0.00 13.79 127826 803126920
dm-3 0.37 0.43 2.51 25255000 146084456
==============
df -k
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
49580256 36870396 10150680 79% /
/dev/mapper/VolGroup00-LogVol03
748313168 660142564 49545324 94% /var
/dev/mapper/VolGroup00-LogVol02
19838052 211072 18602980 2% /tmp
/dev/sda1 295561 24745 255556 9% /boot
tmpfs 12333076 0 12333076 0% /dev/shm
========
cat /proc/meminfo
MemTotal: 24666152 kB
MemFree: 8192640 kB
Buffers: 46512 kB
Cached: 5597864 kB
SwapCached: 183964 kB
Active: 15507332 kB
Inactive: 756136 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 24666152 kB
LowFree: 8192640 kB
SwapTotal: 32767992 kB
SwapFree: 32219520 kB
Dirty: 292 kB
Writeback: 0 kB
AnonPages: 10617088 kB
Mapped: 2175692 kB
Slab: 120216 kB
PageTables: 38128 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 45101068 kB
Committed_AS: 18324080 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 279376 kB
VmallocChunk: 34359458551 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
Hugepagesize: 2048 kB
=========================
* Re: Performance issues
2015-03-13 11:44 Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 12:58 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 13:03 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 13:10 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 19:59 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 20:36 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-13 20:46 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 22:03 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 22:25 ` Re: Performance issues Scott Marlowe <scott.marlowe@gmail.com>
2015-03-13 23:28 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
@ 2015-03-14 00:12 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-14 21:06 ` Re: Performance issues Gavin Flower <GavinFlower@archidevsys.co.nz>
2015-03-16 19:43 ` Re: Performance issues Jim Nasby <Jim.Nasby@BlueTreble.com>
0 siblings, 2 replies; 59+ messages in thread
From: Tomas Vondra @ 2015-03-14 00:12 UTC (permalink / raw)
To: vjoshi@zetainteractive.com, Scott Marlowe <scott.marlowe@gmail.com>; Varadharajan Mukundan <srinathsmn@gmail.com>; +Cc: pgsql-performance
On 14.3.2015 00:28, Vivekanand Joshi wrote:
> Hi Guys,
>
> So here is the full information attached as well as in the link
> provided below:
>
> http://pgsql.privatepaste.com/41207bea45
>
> I can provide new information as well.
Thanks.
We still don't have EXPLAIN ANALYZE - how long was the query running (I
assume it got killed at some point)? It's really difficult to give you
any advices because we don't know where the problem is.
If EXPLAIN ANALYZE really takes too long (say, it does not complete
after an hour / over night), you'll have to break the query into parts
and first tweak those independently.
For example in the first message you mentioned that select from the
S_V_D_CAMPAIGN_HIERARCHY view takes ~9 minutes, so start with that. Give
us EXPLAIN ANALYZE for that query.
Few more comments:
(1) You're using CTEs - be aware that CTEs are not just aliases, but
impact planning / optimization, and in some cases may prevent
proper optimization. Try replacing them with plain views.
(2) Varadharajan Mukundan already recommended you to create index on
s_f_promotion_history.send_dt. Have you tried that? You may also
try creating an index on all the columns needed by the query, so
that "Index Only Scan" is possible.
(3) There are probably additional indexes that might be useful here.
What I'd try is adding indexes on all columns that are either a
foreign key or used in a WHERE condition. This might be an
overkill in some cases, but let's see.
(4) I suspect many of the relations referenced in the views are not
actually needed in the query, i.e. the join is performed but
then it's just discarded because those columns are not used.
Try to simplify the views as much has possible - remove all the
tables that are not really necessary to run the query. If two
queries need different tables, maybe defining two views is
a better approach.
(5) The vmstat / iostat data are pretty useless - what you provided are
averages since the machine was started, but we need a few samples
collected when the query is running. I.e. start the query, and then
give us a few samples from these commands:
iostat -x -k 1
vmstat 1
> Would like to see if queries of these type can actually run in
> postgres server?
Why not? We're running DWH applications on tens/hundreds of GBs.
> If yes, what would be the minimum requirements for hardware? We would
> like to migrate our whole solution on PostgreSQL as we can spend on
> hardware as much as we can but working on a proprietary appliance is
> becoming very difficult for us.
That's difficult to say, because we really don't know where the problem
is and how much the queries can be optimized.
--
Tomas Vondra http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Performance issues
2015-03-13 11:44 Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 12:58 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 13:03 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 13:10 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 19:59 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 20:36 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-13 20:46 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 22:03 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 22:25 ` Re: Performance issues Scott Marlowe <scott.marlowe@gmail.com>
2015-03-13 23:28 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-14 00:12 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
@ 2015-03-14 21:06 ` Gavin Flower <GavinFlower@archidevsys.co.nz>
2015-03-14 21:23 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
1 sibling, 1 reply; 59+ messages in thread
From: Gavin Flower @ 2015-03-14 21:06 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; vjoshi@zetainteractive.com, Scott Marlowe <scott.marlowe@gmail.com>; Varadharajan Mukundan <srinathsmn@gmail.com>; +Cc: pgsql-performance
On 14/03/15 13:12, Tomas Vondra wrote:
> On 14.3.2015 00:28, Vivekanand Joshi wrote:
>> Hi Guys,
>>
>> So here is the full information attached as well as in the link
>> provided below:
>>
>> http://pgsql.privatepaste.com/41207bea45
>>
>> I can provide new information as well.
> Thanks.
>
> We still don't have EXPLAIN ANALYZE - how long was the query running (I
> assume it got killed at some point)? It's really difficult to give you
> any advices because we don't know where the problem is.
>
> If EXPLAIN ANALYZE really takes too long (say, it does not complete
> after an hour / over night), you'll have to break the query into parts
> and first tweak those independently.
>
> For example in the first message you mentioned that select from the
> S_V_D_CAMPAIGN_HIERARCHY view takes ~9 minutes, so start with that. Give
> us EXPLAIN ANALYZE for that query.
>
> Few more comments:
>
> (1) You're using CTEs - be aware that CTEs are not just aliases, but
> impact planning / optimization, and in some cases may prevent
> proper optimization. Try replacing them with plain views.
>
> (2) Varadharajan Mukundan already recommended you to create index on
> s_f_promotion_history.send_dt. Have you tried that? You may also
> try creating an index on all the columns needed by the query, so
> that "Index Only Scan" is possible.
>
> (3) There are probably additional indexes that might be useful here.
> What I'd try is adding indexes on all columns that are either a
> foreign key or used in a WHERE condition. This might be an
> overkill in some cases, but let's see.
>
> (4) I suspect many of the relations referenced in the views are not
> actually needed in the query, i.e. the join is performed but
> then it's just discarded because those columns are not used.
> Try to simplify the views as much has possible - remove all the
> tables that are not really necessary to run the query. If two
> queries need different tables, maybe defining two views is
> a better approach.
>
> (5) The vmstat / iostat data are pretty useless - what you provided are
> averages since the machine was started, but we need a few samples
> collected when the query is running. I.e. start the query, and then
> give us a few samples from these commands:
>
> iostat -x -k 1
> vmstat 1
>
>> Would like to see if queries of these type can actually run in
>> postgres server?
> Why not? We're running DWH applications on tens/hundreds of GBs.
>
>> If yes, what would be the minimum requirements for hardware? We would
>> like to migrate our whole solution on PostgreSQL as we can spend on
>> hardware as much as we can but working on a proprietary appliance is
>> becoming very difficult for us.
> That's difficult to say, because we really don't know where the problem
> is and how much the queries can be optimized.
>
>
I notice that no one appears to have suggested the default setting in
postgresql.conf - these need changing as they are initially set up for
small machines, and to let PostgreSQL take anywhere near full advantage
of a box have large amounts of RAM, you need to change some of the
configuration settings!
For example 'temp_buffers' (default 8MB) and 'maintenance_work_mem'
(default 16MB) should be drastically increased, and there are other
settings that need changing. The precise values depend on many factors,
but the initial values set by default are definitely far too small for
your usage.
Am assuming that you are looking at PostgreSQL 9.4.
Cheers,
Gavin
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Performance issues
2015-03-13 11:44 Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 12:58 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 13:03 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 13:10 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 19:59 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 20:36 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-13 20:46 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 22:03 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 22:25 ` Re: Performance issues Scott Marlowe <scott.marlowe@gmail.com>
2015-03-13 23:28 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-14 00:12 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-14 21:06 ` Re: Performance issues Gavin Flower <GavinFlower@archidevsys.co.nz>
@ 2015-03-14 21:23 ` Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-14 21:32 ` Re: Performance issues Gavin Flower <GavinFlower@archidevsys.co.nz>
0 siblings, 1 reply; 59+ messages in thread
From: Varadharajan Mukundan @ 2015-03-14 21:23 UTC (permalink / raw)
To: Gavin Flower <GavinFlower@archidevsys.co.nz>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; vjoshi@zetainteractive.com, Scott Marlowe <scott.marlowe@gmail.com>; pgsql-performance
Hi Gavin,
Vivekanand is his first mail itself mentioned the below configuration
of postgresql.conf. It looks good enough to me.
Total Memory : 8 GB
shared_buffers = 2GB
work_mem = 64MB
maintenance_work_mem = 700MB
effective_cache_size = 4GB
On Sat, Mar 14, 2015 at 10:06 PM, Gavin Flower
<GavinFlower@archidevsys.co.nz> wrote:
> On 14/03/15 13:12, Tomas Vondra wrote:
>>
>> On 14.3.2015 00:28, Vivekanand Joshi wrote:
>>>
>>> Hi Guys,
>>>
>>> So here is the full information attached as well as in the link
>>> provided below:
>>>
>>> http://pgsql.privatepaste.com/41207bea45
>>>
>>> I can provide new information as well.
>>
>> Thanks.
>>
>> We still don't have EXPLAIN ANALYZE - how long was the query running (I
>> assume it got killed at some point)? It's really difficult to give you
>> any advices because we don't know where the problem is.
>>
>> If EXPLAIN ANALYZE really takes too long (say, it does not complete
>> after an hour / over night), you'll have to break the query into parts
>> and first tweak those independently.
>>
>> For example in the first message you mentioned that select from the
>> S_V_D_CAMPAIGN_HIERARCHY view takes ~9 minutes, so start with that. Give
>> us EXPLAIN ANALYZE for that query.
>>
>> Few more comments:
>>
>> (1) You're using CTEs - be aware that CTEs are not just aliases, but
>> impact planning / optimization, and in some cases may prevent
>> proper optimization. Try replacing them with plain views.
>>
>> (2) Varadharajan Mukundan already recommended you to create index on
>> s_f_promotion_history.send_dt. Have you tried that? You may also
>> try creating an index on all the columns needed by the query, so
>> that "Index Only Scan" is possible.
>>
>> (3) There are probably additional indexes that might be useful here.
>> What I'd try is adding indexes on all columns that are either a
>> foreign key or used in a WHERE condition. This might be an
>> overkill in some cases, but let's see.
>>
>> (4) I suspect many of the relations referenced in the views are not
>> actually needed in the query, i.e. the join is performed but
>> then it's just discarded because those columns are not used.
>> Try to simplify the views as much has possible - remove all the
>> tables that are not really necessary to run the query. If two
>> queries need different tables, maybe defining two views is
>> a better approach.
>>
>> (5) The vmstat / iostat data are pretty useless - what you provided are
>> averages since the machine was started, but we need a few samples
>> collected when the query is running. I.e. start the query, and then
>> give us a few samples from these commands:
>>
>> iostat -x -k 1
>> vmstat 1
>>
>>> Would like to see if queries of these type can actually run in
>>> postgres server?
>>
>> Why not? We're running DWH applications on tens/hundreds of GBs.
>>
>>> If yes, what would be the minimum requirements for hardware? We would
>>> like to migrate our whole solution on PostgreSQL as we can spend on
>>> hardware as much as we can but working on a proprietary appliance is
>>> becoming very difficult for us.
>>
>> That's difficult to say, because we really don't know where the problem
>> is and how much the queries can be optimized.
>>
>>
> I notice that no one appears to have suggested the default setting in
> postgresql.conf - these need changing as they are initially set up for small
> machines, and to let PostgreSQL take anywhere near full advantage of a box
> have large amounts of RAM, you need to change some of the configuration
> settings!
>
> For example 'temp_buffers' (default 8MB) and 'maintenance_work_mem' (default
> 16MB) should be drastically increased, and there are other settings that
> need changing. The precise values depend on many factors, but the initial
> values set by default are definitely far too small for your usage.
>
> Am assuming that you are looking at PostgreSQL 9.4.
>
>
>
> Cheers,
> Gavin
>
>
--
Thanks,
M. Varadharajan
------------------------------------------------
"Experience is what you get when you didn't get what you wanted"
-By Prof. Randy Pausch in "The Last Lecture"
My Journal :- www.thinkasgeek.wordpress.com
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Performance issues
2015-03-13 11:44 Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 12:58 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 13:03 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 13:10 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 19:59 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 20:36 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-13 20:46 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 22:03 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 22:25 ` Re: Performance issues Scott Marlowe <scott.marlowe@gmail.com>
2015-03-13 23:28 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-14 00:12 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-14 21:06 ` Re: Performance issues Gavin Flower <GavinFlower@archidevsys.co.nz>
2015-03-14 21:23 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
@ 2015-03-14 21:32 ` Gavin Flower <GavinFlower@archidevsys.co.nz>
2015-03-16 12:24 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
0 siblings, 1 reply; 59+ messages in thread
From: Gavin Flower @ 2015-03-14 21:32 UTC (permalink / raw)
To: Varadharajan Mukundan <srinathsmn@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; vjoshi@zetainteractive.com, Scott Marlowe <scott.marlowe@gmail.com>; pgsql-performance
On 15/03/15 10:23, Varadharajan Mukundan wrote:
> Hi Gavin,
>
> Vivekanand is his first mail itself mentioned the below configuration
> of postgresql.conf. It looks good enough to me.
>
> Total Memory : 8 GB
>
> shared_buffers = 2GB
>
> work_mem = 64MB
>
> maintenance_work_mem = 700MB
>
> effective_cache_size = 4GB
Sorry, it didn't register when I read it!
(Probably reading too fast)
>
> On Sat, Mar 14, 2015 at 10:06 PM, Gavin Flower
> <GavinFlower@archidevsys.co.nz> wrote:
>> On 14/03/15 13:12, Tomas Vondra wrote:
>>> On 14.3.2015 00:28, Vivekanand Joshi wrote:
>>>> Hi Guys,
>>>>
>>>> So here is the full information attached as well as in the link
>>>> provided below:
>>>>
>>>> http://pgsql.privatepaste.com/41207bea45
>>>>
>>>> I can provide new information as well.
>>> Thanks.
>>>
>>> We still don't have EXPLAIN ANALYZE - how long was the query running (I
>>> assume it got killed at some point)? It's really difficult to give you
>>> any advices because we don't know where the problem is.
>>>
>>> If EXPLAIN ANALYZE really takes too long (say, it does not complete
>>> after an hour / over night), you'll have to break the query into parts
>>> and first tweak those independently.
>>>
>>> For example in the first message you mentioned that select from the
>>> S_V_D_CAMPAIGN_HIERARCHY view takes ~9 minutes, so start with that. Give
>>> us EXPLAIN ANALYZE for that query.
>>>
>>> Few more comments:
>>>
>>> (1) You're using CTEs - be aware that CTEs are not just aliases, but
>>> impact planning / optimization, and in some cases may prevent
>>> proper optimization. Try replacing them with plain views.
>>>
>>> (2) Varadharajan Mukundan already recommended you to create index on
>>> s_f_promotion_history.send_dt. Have you tried that? You may also
>>> try creating an index on all the columns needed by the query, so
>>> that "Index Only Scan" is possible.
>>>
>>> (3) There are probably additional indexes that might be useful here.
>>> What I'd try is adding indexes on all columns that are either a
>>> foreign key or used in a WHERE condition. This might be an
>>> overkill in some cases, but let's see.
>>>
>>> (4) I suspect many of the relations referenced in the views are not
>>> actually needed in the query, i.e. the join is performed but
>>> then it's just discarded because those columns are not used.
>>> Try to simplify the views as much has possible - remove all the
>>> tables that are not really necessary to run the query. If two
>>> queries need different tables, maybe defining two views is
>>> a better approach.
>>>
>>> (5) The vmstat / iostat data are pretty useless - what you provided are
>>> averages since the machine was started, but we need a few samples
>>> collected when the query is running. I.e. start the query, and then
>>> give us a few samples from these commands:
>>>
>>> iostat -x -k 1
>>> vmstat 1
>>>
>>>> Would like to see if queries of these type can actually run in
>>>> postgres server?
>>> Why not? We're running DWH applications on tens/hundreds of GBs.
>>>
>>>> If yes, what would be the minimum requirements for hardware? We would
>>>> like to migrate our whole solution on PostgreSQL as we can spend on
>>>> hardware as much as we can but working on a proprietary appliance is
>>>> becoming very difficult for us.
>>> That's difficult to say, because we really don't know where the problem
>>> is and how much the queries can be optimized.
>>>
>>>
>> I notice that no one appears to have suggested the default setting in
>> postgresql.conf - these need changing as they are initially set up for small
>> machines, and to let PostgreSQL take anywhere near full advantage of a box
>> have large amounts of RAM, you need to change some of the configuration
>> settings!
>>
>> For example 'temp_buffers' (default 8MB) and 'maintenance_work_mem' (default
>> 16MB) should be drastically increased, and there are other settings that
>> need changing. The precise values depend on many factors, but the initial
>> values set by default are definitely far too small for your usage.
>>
>> Am assuming that you are looking at PostgreSQL 9.4.
>>
>>
>>
>> Cheers,
>> Gavin
>>
>>
>
>
>
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Performance issues
2015-03-13 11:44 Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 12:58 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 13:03 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 13:10 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 19:59 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 20:36 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-13 20:46 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 22:03 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 22:25 ` Re: Performance issues Scott Marlowe <scott.marlowe@gmail.com>
2015-03-13 23:28 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-14 00:12 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-14 21:06 ` Re: Performance issues Gavin Flower <GavinFlower@archidevsys.co.nz>
2015-03-14 21:23 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-14 21:32 ` Re: Performance issues Gavin Flower <GavinFlower@archidevsys.co.nz>
@ 2015-03-16 12:24 ` Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-16 17:49 ` Re: Performance issues Marc Mamin <M.Mamin@intershop.de>
0 siblings, 1 reply; 59+ messages in thread
From: Vivekanand Joshi @ 2015-03-16 12:24 UTC (permalink / raw)
To: Gavin Flower <GavinFlower@archidevsys.co.nz>; Varadharajan Mukundan <srinathsmn@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>; pgsql-performance
Hi Team,
This is the EXPLAIN ANALYZE for one of the view : S_V_D_CAMPAIGN_HIERARCHY:
===========================================
Nested Loop (cost=33666.96..37971.39 rows=1 width=894) (actual
time=443.556..966558.767 rows=45360 loops=1)
Join Filter: (tp_exec.touchpoint_execution_id =
valid_executions.touchpoint_execution_id)
Rows Removed by Join Filter: 3577676116
CTE valid_executions
-> Hash Join (cost=13753.53..31711.17 rows=1 width=8) (actual
time=232.571..357.749 rows=52997 loops=1)
Hash Cond:
((s_f_touchpoint_execution_status_history_1.touchpoint_execution_id =
s_f_touchpoint_execution_status_history.touchpoint_execution_id) AND ((max(s
_f_touchpoint_execution_status_history_1.creation_dt)) =
s_f_touchpoint_execution_status_history.creation_dt))
-> HashAggregate (cost=6221.56..6905.66 rows=68410 width=16)
(actual time=139.713..171.340 rows=76454 loops=1)
-> Seq Scan on s_f_touchpoint_execution_status_history
s_f_touchpoint_execution_status_history_1 (cost=0.00..4766.04 rows=291104
width=16) (actual ti
me=0.006..38.582 rows=291104 loops=1)
-> Hash (cost=5493.80..5493.80 rows=135878 width=16) (actual
time=92.737..92.737 rows=136280 loops=1)
Buckets: 16384 Batches: 1 Memory Usage: 6389kB
-> Seq Scan on s_f_touchpoint_execution_status_history
(cost=0.00..5493.80 rows=135878 width=16) (actual time=0.012..55.078
rows=136280 loops=1)
Filter: (touchpoint_execution_status_type_id = ANY
('{3,4}'::integer[]))
Rows Removed by Filter: 154824
-> Nested Loop Left Join (cost=1955.80..6260.19 rows=1 width=894)
(actual time=31.608..3147.015 rows=67508 loops=1)
-> Nested Loop (cost=1955.67..6260.04 rows=1 width=776) (actual
time=31.602..2912.625 rows=67508 loops=1)
-> Nested Loop Left Join (cost=1955.54..6259.87 rows=1
width=658) (actual time=31.595..2713.696 rows=72427 loops=1)
-> Nested Loop Left Join (cost=1955.40..6259.71
rows=1 width=340) (actual time=31.589..2532.926 rows=72427 loops=1)
-> Nested Loop Left Join (cost=1955.27..6259.55
rows=1 width=222) (actual time=31.581..2354.662 rows=72427 loops=1)
-> Nested Loop (cost=1954.99..6259.24
rows=1 width=197) (actual time=31.572..2090.104 rows=72427 loops=1)
-> Nested Loop
(cost=1954.71..6258.92 rows=1 width=173) (actual time=31.562..1802.857
rows=72427 loops=1)
Join Filter:
(camp_exec.campaign_id = wave.campaign_id)
Rows Removed by Join Filter:
243
-> Nested Loop
(cost=1954.42..6254.67 rows=13 width=167) (actual time=31.551..1468.718
rows=72670 loops=1)
-> Hash Join
(cost=1954.13..6249.67 rows=13 width=108) (actual time=31.525..402.039
rows=72670 loops=1)
Hash Cond:
((tp_exec.touchpoint_id = tp.touchpoint_id) AND (wave_exec.wave_id =
tp.wave_id))
-> Hash Join
(cost=1576.83..4595.51 rows=72956 width=90) (actual time=26.254..256.328
rows=72956 loops=1)
Hash Cond:
(tp_exec.wave_execution_id = wave_exec.wave_execution_id)
-> Seq Scan
on s_d_touchpoint_execution tp_exec (cost=0.00..1559.56 rows=72956
width=42) (actual time=0.005..76.099 rows=72956 loops=1)
-> Hash
(cost=1001.37..1001.37 rows=46037 width=56) (actual time=26.178..26.178
rows=46037 loops=1)
Buckets:
8192 Batches: 1 Memory Usage: 4104kB
-> Seq
Scan on s_d_wave_execution wave_exec (cost=0.00..1001.37 rows=46037
width=56) (actual time=0.006..10.388 rows=46037 loops=1)
-> Hash
(cost=212.72..212.72 rows=10972 width=26) (actual time=5.252..5.252
rows=10972 loops=1)
Buckets: 2048
Batches: 1 Memory Usage: 645kB
-> Seq Scan
on s_d_touchpoint tp (cost=0.00..212.72 rows=10972 width=26) (actual
time=0.012..2.319 rows=10972 loops=1)
-> Index Scan using
s_d_campaign_execution_idx on s_d_campaign_execution camp_exec
(cost=0.29..0.37 rows=1 width=67) (actual time=0.013..0.013 rows=1
loops=72670)
Index Cond:
(campaign_execution_id = wave_exec.campaign_execution_id)
-> Index Scan using
s_d_wave_pkey on s_d_wave wave (cost=0.29..0.31 rows=1 width=22) (actual
time=0.003..0.003 rows=1 loops=72670)
Index Cond: (wave_id =
wave_exec.wave_id)
-> Index Scan using
s_d_campaign_pkey on s_d_campaign camp (cost=0.29..0.32 rows=1 width=40)
(actual time=0.003..0.003 rows=1 loops=72427)
Index Cond: (campaign_id =
camp_exec.campaign_id)
-> Index Scan using s_d_content_pkey on
s_d_content content (cost=0.28..0.30 rows=1 width=33) (actual
time=0.002..0.003 rows=1 loops=72427)
Index Cond: (tp_exec.content_id =
content_id)
-> Index Scan using s_d_message_type_pkey on
s_d_message_type message_type (cost=0.13..0.15 rows=1 width=120) (actual
time=0.001..0.002 rows=1 loops=72427)
Index Cond: (tp_exec.message_type_id =
message_type_id)
-> Index Scan using s_d_group_pkey on s_d_group grup
(cost=0.13..0.15 rows=1 width=320) (actual time=0.001..0.002 rows=1
loops=72427)
Index Cond: (camp_exec.group_id = group_id)
-> Index Scan using d_channel_pk on s_d_channel_type channel
(cost=0.13..0.15 rows=1 width=120) (actual time=0.001..0.002 rows=1
loops=72427)
Index Cond: (channel_type_id = tp.channel_type_id)
-> Index Scan using s_d_category_pkey on s_d_category "CATEGORY"
(cost=0.13..0.15 rows=1 width=120) (actual time=0.001..0.002 rows=1
loops=67508)
Index Cond: (camp.category_id = category_id)
-> CTE Scan on valid_executions (cost=0.00..0.02 rows=1 width=8)
(actual time=0.004..6.803 rows=52997 loops=67508)
Total runtime: 966566.574 ms
========================================================
Can you please see it an let me know where is the issue?
-----Original Message-----
From: Gavin Flower [mailto:GavinFlower@archidevsys.co.nz]
Sent: Sunday, March 15, 2015 3:02 AM
To: Varadharajan Mukundan
Cc: Tomas Vondra; vjoshi@zetainteractive.com; Scott Marlowe;
pgsql-performance@postgresql.org
Subject: Re: [PERFORM] Performance issues
On 15/03/15 10:23, Varadharajan Mukundan wrote:
> Hi Gavin,
>
> Vivekanand is his first mail itself mentioned the below configuration
> of postgresql.conf. It looks good enough to me.
>
> Total Memory : 8 GB
>
> shared_buffers = 2GB
>
> work_mem = 64MB
>
> maintenance_work_mem = 700MB
>
> effective_cache_size = 4GB
Sorry, it didn't register when I read it!
(Probably reading too fast)
>
> On Sat, Mar 14, 2015 at 10:06 PM, Gavin Flower
> <GavinFlower@archidevsys.co.nz> wrote:
>> On 14/03/15 13:12, Tomas Vondra wrote:
>>> On 14.3.2015 00:28, Vivekanand Joshi wrote:
>>>> Hi Guys,
>>>>
>>>> So here is the full information attached as well as in the link
>>>> provided below:
>>>>
>>>> http://pgsql.privatepaste.com/41207bea45
>>>>
>>>> I can provide new information as well.
>>> Thanks.
>>>
>>> We still don't have EXPLAIN ANALYZE - how long was the query running
>>> (I assume it got killed at some point)? It's really difficult to
>>> give you any advices because we don't know where the problem is.
>>>
>>> If EXPLAIN ANALYZE really takes too long (say, it does not complete
>>> after an hour / over night), you'll have to break the query into
>>> parts and first tweak those independently.
>>>
>>> For example in the first message you mentioned that select from the
>>> S_V_D_CAMPAIGN_HIERARCHY view takes ~9 minutes, so start with that.
>>> Give us EXPLAIN ANALYZE for that query.
>>>
>>> Few more comments:
>>>
>>> (1) You're using CTEs - be aware that CTEs are not just aliases, but
>>> impact planning / optimization, and in some cases may prevent
>>> proper optimization. Try replacing them with plain views.
>>>
>>> (2) Varadharajan Mukundan already recommended you to create index on
>>> s_f_promotion_history.send_dt. Have you tried that? You may also
>>> try creating an index on all the columns needed by the query, so
>>> that "Index Only Scan" is possible.
>>>
>>> (3) There are probably additional indexes that might be useful here.
>>> What I'd try is adding indexes on all columns that are either a
>>> foreign key or used in a WHERE condition. This might be an
>>> overkill in some cases, but let's see.
>>>
>>> (4) I suspect many of the relations referenced in the views are not
>>> actually needed in the query, i.e. the join is performed but
>>> then it's just discarded because those columns are not used.
>>> Try to simplify the views as much has possible - remove all the
>>> tables that are not really necessary to run the query. If two
>>> queries need different tables, maybe defining two views is
>>> a better approach.
>>>
>>> (5) The vmstat / iostat data are pretty useless - what you provided are
>>> averages since the machine was started, but we need a few samples
>>> collected when the query is running. I.e. start the query, and
>>> then
>>> give us a few samples from these commands:
>>>
>>> iostat -x -k 1
>>> vmstat 1
>>>
>>>> Would like to see if queries of these type can actually run in
>>>> postgres server?
>>> Why not? We're running DWH applications on tens/hundreds of GBs.
>>>
>>>> If yes, what would be the minimum requirements for hardware? We
>>>> would like to migrate our whole solution on PostgreSQL as we can
>>>> spend on hardware as much as we can but working on a proprietary
>>>> appliance is becoming very difficult for us.
>>> That's difficult to say, because we really don't know where the
>>> problem is and how much the queries can be optimized.
>>>
>>>
>> I notice that no one appears to have suggested the default setting in
>> postgresql.conf - these need changing as they are initially set up
>> for small machines, and to let PostgreSQL take anywhere near full
>> advantage of a box have large amounts of RAM, you need to change some
>> of the configuration settings!
>>
>> For example 'temp_buffers' (default 8MB) and 'maintenance_work_mem'
>> (default
>> 16MB) should be drastically increased, and there are other settings
>> that need changing. The precise values depend on many factors, but
>> the initial values set by default are definitely far too small for your
>> usage.
>>
>> Am assuming that you are looking at PostgreSQL 9.4.
>>
>>
>>
>> Cheers,
>> Gavin
>>
>>
>
>
>
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Performance issues
2015-03-13 11:44 Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 12:58 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 13:03 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 13:10 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 19:59 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 20:36 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-13 20:46 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 22:03 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 22:25 ` Re: Performance issues Scott Marlowe <scott.marlowe@gmail.com>
2015-03-13 23:28 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-14 00:12 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-14 21:06 ` Re: Performance issues Gavin Flower <GavinFlower@archidevsys.co.nz>
2015-03-14 21:23 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-14 21:32 ` Re: Performance issues Gavin Flower <GavinFlower@archidevsys.co.nz>
2015-03-16 12:24 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
@ 2015-03-16 17:49 ` Marc Mamin <M.Mamin@intershop.de>
2015-03-16 18:24 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 1 reply; 59+ messages in thread
From: Marc Mamin @ 2015-03-16 17:49 UTC (permalink / raw)
To: vjoshi@zetainteractive.com <vjoshi@zetainteractive.com>; +Cc: pgsql-performance
>Hi Team,
>
>This is the EXPLAIN ANALYZE for one of the view : S_V_D_CAMPAIGN_HIERARCHY:
>Rows Removed by Join Filter: 3577676116
That's quite a lot.
You're possibly missing a clause in a join, resulting in a cross join.
It is also helpful to put your result here:
http://explain.depesz.com/
regards,
Marc Mamin
>
>===========================================
>
>
>Nested Loop (cost=33666.96..37971.39 rows=1 width=894) (actual
>time=443.556..966558.767 rows=45360 loops=1)
> Join Filter: (tp_exec.touchpoint_execution_id =
>valid_executions.touchpoint_execution_id)
> Rows Removed by Join Filter: 3577676116
> CTE valid_executions
> -> Hash Join (cost=13753.53..31711.17 rows=1 width=8) (actual
>time=232.571..357.749 rows=52997 loops=1)
> Hash Cond:
>((s_f_touchpoint_execution_status_history_1.touchpoint_execution_id =
>s_f_touchpoint_execution_status_history.touchpoint_execution_id) AND ((max(s
>_f_touchpoint_execution_status_history_1.creation_dt)) =
>s_f_touchpoint_execution_status_history.creation_dt))
> -> HashAggregate (cost=6221.56..6905.66 rows=68410 width=16)
>(actual time=139.713..171.340 rows=76454 loops=1)
> -> Seq Scan on s_f_touchpoint_execution_status_history
>s_f_touchpoint_execution_status_history_1 (cost=0.00..4766.04 rows=291104
>width=16) (actual ti
>me=0.006..38.582 rows=291104 loops=1)
> -> Hash (cost=5493.80..5493.80 rows=135878 width=16) (actual
>time=92.737..92.737 rows=136280 loops=1)
> Buckets: 16384 Batches: 1 Memory Usage: 6389kB
> -> Seq Scan on s_f_touchpoint_execution_status_history
>(cost=0.00..5493.80 rows=135878 width=16) (actual time=0.012..55.078
>rows=136280 loops=1)
> Filter: (touchpoint_execution_status_type_id = ANY
>('{3,4}'::integer[]))
> Rows Removed by Filter: 154824
> -> Nested Loop Left Join (cost=1955.80..6260.19 rows=1 width=894)
>(actual time=31.608..3147.015 rows=67508 loops=1)
> -> Nested Loop (cost=1955.67..6260.04 rows=1 width=776) (actual
>time=31.602..2912.625 rows=67508 loops=1)
> -> Nested Loop Left Join (cost=1955.54..6259.87 rows=1
>width=658) (actual time=31.595..2713.696 rows=72427 loops=1)
> -> Nested Loop Left Join (cost=1955.40..6259.71
>rows=1 width=340) (actual time=31.589..2532.926 rows=72427 loops=1)
> -> Nested Loop Left Join (cost=1955.27..6259.55
>rows=1 width=222) (actual time=31.581..2354.662 rows=72427 loops=1)
> -> Nested Loop (cost=1954.99..6259.24
>rows=1 width=197) (actual time=31.572..2090.104 rows=72427 loops=1)
> -> Nested Loop
>(cost=1954.71..6258.92 rows=1 width=173) (actual time=31.562..1802.857
>rows=72427 loops=1)
> Join Filter:
>(camp_exec.campaign_id = wave.campaign_id)
> Rows Removed by Join Filter:
>243
> -> Nested Loop
>(cost=1954.42..6254.67 rows=13 width=167) (actual time=31.551..1468.718
>rows=72670 loops=1)
> -> Hash Join
>(cost=1954.13..6249.67 rows=13 width=108) (actual time=31.525..402.039
>rows=72670 loops=1)
> Hash Cond:
>((tp_exec.touchpoint_id = tp.touchpoint_id) AND (wave_exec.wave_id =
>tp.wave_id))
> -> Hash Join
>(cost=1576.83..4595.51 rows=72956 width=90) (actual time=26.254..256.328
>rows=72956 loops=1)
> Hash Cond:
>(tp_exec.wave_execution_id = wave_exec.wave_execution_id)
> -> Seq Scan
>on s_d_touchpoint_execution tp_exec (cost=0.00..1559.56 rows=72956
>width=42) (actual time=0.005..76.099 rows=72956 loops=1)
> -> Hash
>(cost=1001.37..1001.37 rows=46037 width=56) (actual time=26.178..26.178
>rows=46037 loops=1)
> Buckets:
>8192 Batches: 1 Memory Usage: 4104kB
> -> Seq
>Scan on s_d_wave_execution wave_exec (cost=0.00..1001.37 rows=46037
>width=56) (actual time=0.006..10.388 rows=46037 loops=1)
> -> Hash
>(cost=212.72..212.72 rows=10972 width=26) (actual time=5.252..5.252
>rows=10972 loops=1)
> Buckets: 2048
>Batches: 1 Memory Usage: 645kB
> -> Seq Scan
>on s_d_touchpoint tp (cost=0.00..212.72 rows=10972 width=26) (actual
>time=0.012..2.319 rows=10972 loops=1)
> -> Index Scan using
>s_d_campaign_execution_idx on s_d_campaign_execution camp_exec
>(cost=0.29..0.37 rows=1 width=67) (actual time=0.013..0.013 rows=1
>loops=72670)
> Index Cond:
>(campaign_execution_id = wave_exec.campaign_execution_id)
> -> Index Scan using
>s_d_wave_pkey on s_d_wave wave (cost=0.29..0.31 rows=1 width=22) (actual
>time=0.003..0.003 rows=1 loops=72670)
> Index Cond: (wave_id =
>wave_exec.wave_id)
> -> Index Scan using
>s_d_campaign_pkey on s_d_campaign camp (cost=0.29..0.32 rows=1 width=40)
>(actual time=0.003..0.003 rows=1 loops=72427)
> Index Cond: (campaign_id =
>camp_exec.campaign_id)
> -> Index Scan using s_d_content_pkey on
>s_d_content content (cost=0.28..0.30 rows=1 width=33) (actual
>time=0.002..0.003 rows=1 loops=72427)
> Index Cond: (tp_exec.content_id =
>content_id)
> -> Index Scan using s_d_message_type_pkey on
>s_d_message_type message_type (cost=0.13..0.15 rows=1 width=120) (actual
>time=0.001..0.002 rows=1 loops=72427)
> Index Cond: (tp_exec.message_type_id =
>message_type_id)
> -> Index Scan using s_d_group_pkey on s_d_group grup
>(cost=0.13..0.15 rows=1 width=320) (actual time=0.001..0.002 rows=1
>loops=72427)
> Index Cond: (camp_exec.group_id = group_id)
> -> Index Scan using d_channel_pk on s_d_channel_type channel
>(cost=0.13..0.15 rows=1 width=120) (actual time=0.001..0.002 rows=1
>loops=72427)
> Index Cond: (channel_type_id = tp.channel_type_id)
> -> Index Scan using s_d_category_pkey on s_d_category "CATEGORY"
>(cost=0.13..0.15 rows=1 width=120) (actual time=0.001..0.002 rows=1
>loops=67508)
> Index Cond: (camp.category_id = category_id)
> -> CTE Scan on valid_executions (cost=0.00..0.02 rows=1 width=8)
>(actual time=0.004..6.803 rows=52997 loops=67508)
> Total runtime: 966566.574 ms
>
>========================================================
>
>Can you please see it an let me know where is the issue?
>
>
>-----Original Message-----
>From: Gavin Flower [mailto:GavinFlower@archidevsys.co.nz]
>Sent: Sunday, March 15, 2015 3:02 AM
>To: Varadharajan Mukundan
>Cc: Tomas Vondra; vjoshi@zetainteractive.com; Scott Marlowe;
>pgsql-performance@postgresql.org
>Subject: Re: [PERFORM] Performance issues
>
>On 15/03/15 10:23, Varadharajan Mukundan wrote:
>> Hi Gavin,
>>
>> Vivekanand is his first mail itself mentioned the below configuration
>> of postgresql.conf. It looks good enough to me.
>>
>> Total Memory : 8 GB
>>
>> shared_buffers = 2GB
>>
>> work_mem = 64MB
>>
>> maintenance_work_mem = 700MB
>>
>> effective_cache_size = 4GB
>
>
>Sorry, it didn't register when I read it!
>(Probably reading too fast)
>>
>> On Sat, Mar 14, 2015 at 10:06 PM, Gavin Flower
>> <GavinFlower@archidevsys.co.nz> wrote:
>>> On 14/03/15 13:12, Tomas Vondra wrote:
>>>> On 14.3.2015 00:28, Vivekanand Joshi wrote:
>>>>> Hi Guys,
>>>>>
>>>>> So here is the full information attached as well as in the link
>>>>> provided below:
>>>>>
>>>>> http://pgsql.privatepaste.com/41207bea45
>>>>>
>>>>> I can provide new information as well.
>>>> Thanks.
>>>>
>>>> We still don't have EXPLAIN ANALYZE - how long was the query running
>>>> (I assume it got killed at some point)? It's really difficult to
>>>> give you any advices because we don't know where the problem is.
>>>>
>>>> If EXPLAIN ANALYZE really takes too long (say, it does not complete
>>>> after an hour / over night), you'll have to break the query into
>>>> parts and first tweak those independently.
>>>>
>>>> For example in the first message you mentioned that select from the
>>>> S_V_D_CAMPAIGN_HIERARCHY view takes ~9 minutes, so start with that.
>>>> Give us EXPLAIN ANALYZE for that query.
>>>>
>>>> Few more comments:
>>>>
>>>> (1) You're using CTEs - be aware that CTEs are not just aliases, but
>>>> impact planning / optimization, and in some cases may prevent
>>>> proper optimization. Try replacing them with plain views.
>>>>
>>>> (2) Varadharajan Mukundan already recommended you to create index on
>>>> s_f_promotion_history.send_dt. Have you tried that? You may also
>>>> try creating an index on all the columns needed by the query, so
>>>> that "Index Only Scan" is possible.
>>>>
>>>> (3) There are probably additional indexes that might be useful here.
>>>> What I'd try is adding indexes on all columns that are either a
>>>> foreign key or used in a WHERE condition. This might be an
>>>> overkill in some cases, but let's see.
>>>>
>>>> (4) I suspect many of the relations referenced in the views are not
>>>> actually needed in the query, i.e. the join is performed but
>>>> then it's just discarded because those columns are not used.
>>>> Try to simplify the views as much has possible - remove all the
>>>> tables that are not really necessary to run the query. If two
>>>> queries need different tables, maybe defining two views is
>>>> a better approach.
>>>>
>>>> (5) The vmstat / iostat data are pretty useless - what you provided are
>>>> averages since the machine was started, but we need a few samples
>>>> collected when the query is running. I.e. start the query, and
>>>> then
>>>> give us a few samples from these commands:
>>>>
>>>> iostat -x -k 1
>>>> vmstat 1
>>>>
>>>>> Would like to see if queries of these type can actually run in
>>>>> postgres server?
>>>> Why not? We're running DWH applications on tens/hundreds of GBs.
>>>>
>>>>> If yes, what would be the minimum requirements for hardware? We
>>>>> would like to migrate our whole solution on PostgreSQL as we can
>>>>> spend on hardware as much as we can but working on a proprietary
>>>>> appliance is becoming very difficult for us.
>>>> That's difficult to say, because we really don't know where the
>>>> problem is and how much the queries can be optimized.
>>>>
>>>>
>>> I notice that no one appears to have suggested the default setting in
>>> postgresql.conf - these need changing as they are initially set up
>>> for small machines, and to let PostgreSQL take anywhere near full
>>> advantage of a box have large amounts of RAM, you need to change some
>>> of the configuration settings!
>>>
>>> For example 'temp_buffers' (default 8MB) and 'maintenance_work_mem'
>>> (default
>>> 16MB) should be drastically increased, and there are other settings
>>> that need changing. The precise values depend on many factors, but
>>> the initial values set by default are definitely far too small for your
>>> usage.
>>>
>>> Am assuming that you are looking at PostgreSQL 9.4.
>>>
>>>
>>>
>>> Cheers,
>>> Gavin
>>>
>>>
>>
>>
>>
>
>
>--
>Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
>To make changes to your subscription:
>http://www.postgresql.org/mailpref/pgsql-performance
>
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Performance issues
2015-03-13 11:44 Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 12:58 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 13:03 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 13:10 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 19:59 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 20:36 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-13 20:46 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 22:03 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 22:25 ` Re: Performance issues Scott Marlowe <scott.marlowe@gmail.com>
2015-03-13 23:28 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-14 00:12 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-14 21:06 ` Re: Performance issues Gavin Flower <GavinFlower@archidevsys.co.nz>
2015-03-14 21:23 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-14 21:32 ` Re: Performance issues Gavin Flower <GavinFlower@archidevsys.co.nz>
2015-03-16 12:24 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-16 17:49 ` Re: Performance issues Marc Mamin <M.Mamin@intershop.de>
@ 2015-03-16 18:24 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-16 18:32 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
0 siblings, 1 reply; 59+ messages in thread
From: Tomas Vondra @ 2015-03-16 18:24 UTC (permalink / raw)
To: pgsql-performance
On 16.3.2015 18:49, Marc Mamin wrote:
>
>> Hi Team,
>>
>> This is the EXPLAIN ANALYZE for one of the view : S_V_D_CAMPAIGN_HIERARCHY:
FWIW, this is a somewhat more readable version of the plan:
http://explain.depesz.com/s/nbB
In the future, please do two things:
(1) Attach the plan as a text file, because the mail clients tend to
screw things up (wrapping long lines). Unless the plan is trivial,
of course - but pgsql-performance usually deals with complex stuff.
(2) Put the plan on explain.depesz.com helps too, because it's
considerably more readable (but always do 1, because resorces
placed somewhere else tends to disappear, and the posts then make
very little sense, which is bad when searching in the archives)
(3) Same for stuff pasted somewhere else - always attach it to the
message. For example I'd like to give you more accurate advice, but
I can't as http://pgsql.privatepaste.com/41207bea45 is unavailable.
>
>
> >Rows Removed by Join Filter: 3577676116
>
> That's quite a lot.
> You're possibly missing a clause in a join, resulting in a cross join.
> It is also helpful to put your result here:
> http://explain.depesz.com/
> regards,
IMHO this is merely a consequence of using the CTE, which produces 52997
rows and is scanned 67508x as the inner relation of a nested loop. That
gives you 3577721476 tuples in total, and only 45360 are kept (hence
3577676116 are removed).
This is a prime example of why CTEs are not just aliases for subqueries,
but may actually cause serious trouble.
There are other issues (e.g. the row count estimate of the CTE is
seriously off, most likely because of the HashAggregate in the outer
branch), but that's a secondary issue IMHO.
Vivekanand, try this (in the order of intrusiveness):
(1) Get rid of the CTE, and just replace it with subselect in the FROM
part of the query, so instead of this:
WITH valid_executions AS (...)
SELECT ... FROM ... JOIN valid_executions ON (...)
you'll have something like this:
SELECT ... FROM ... JOIN (...) AS valid_executions ON (...)
This way the subselect will optimized properly.
(2) Replace the CTE with a materialized view, or a temporary table.
This has both advantages and disadvantages - the main advantage is
that you can create indexes, collect statistics. Disadvantage is
you have to refresh the MV, fill temporary table etc.
I expect (1) to improve the performance significantly, and (2) might
improve it even further by fixing the misestimates.
regards
--
Tomas Vondra http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Performance issues
2015-03-13 11:44 Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 12:58 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 13:03 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 13:10 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 19:59 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 20:36 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-13 20:46 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 22:03 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 22:25 ` Re: Performance issues Scott Marlowe <scott.marlowe@gmail.com>
2015-03-13 23:28 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-14 00:12 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-14 21:06 ` Re: Performance issues Gavin Flower <GavinFlower@archidevsys.co.nz>
2015-03-14 21:23 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-14 21:32 ` Re: Performance issues Gavin Flower <GavinFlower@archidevsys.co.nz>
2015-03-16 12:24 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-16 17:49 ` Re: Performance issues Marc Mamin <M.Mamin@intershop.de>
2015-03-16 18:24 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
@ 2015-03-16 18:32 ` Vivekanand Joshi <vjoshi@zetainteractive.com>
0 siblings, 0 replies; 59+ messages in thread
From: Vivekanand Joshi @ 2015-03-16 18:32 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: pgsql-performance
Hey guys, thanks a lot.
This is really helping. I am learning a lot. BTW, I changed CTE into
subquery and it improved the performance by miles. I am getting the result
in less than 3 seconds, though I am using a 24 GB ram server. It is still a
great turnaround time as compared to previous execution time.
Now I will look into the bigger query. I read explain analyze and that
helped a lot. I will be coming up with more questions tomorrow as bigger
query still has got some problems.
On 16 Mar 2015 23:55, "Tomas Vondra" <tomas.vondra@2ndquadrant.com> wrote:
> On 16.3.2015 18:49, Marc Mamin wrote:
> >
> >> Hi Team,
> >>
> >> This is the EXPLAIN ANALYZE for one of the view :
> S_V_D_CAMPAIGN_HIERARCHY:
>
> FWIW, this is a somewhat more readable version of the plan:
>
> http://explain.depesz.com/s/nbB
>
> In the future, please do two things:
>
> (1) Attach the plan as a text file, because the mail clients tend to
> screw things up (wrapping long lines). Unless the plan is trivial,
> of course - but pgsql-performance usually deals with complex stuff.
>
> (2) Put the plan on explain.depesz.com helps too, because it's
> considerably more readable (but always do 1, because resorces
> placed somewhere else tends to disappear, and the posts then make
> very little sense, which is bad when searching in the archives)
>
> (3) Same for stuff pasted somewhere else - always attach it to the
> message. For example I'd like to give you more accurate advice, but
> I can't as http://pgsql.privatepaste.com/41207bea45 is unavailable.
>
> >
> >
> > >Rows Removed by Join Filter: 3577676116
> >
> > That's quite a lot.
> > You're possibly missing a clause in a join, resulting in a cross
> join.
> > It is also helpful to put your result here:
> > http://explain.depesz.com/
> > regards,
>
> IMHO this is merely a consequence of using the CTE, which produces 52997
> rows and is scanned 67508x as the inner relation of a nested loop. That
> gives you 3577721476 tuples in total, and only 45360 are kept (hence
> 3577676116 are removed).
>
> This is a prime example of why CTEs are not just aliases for subqueries,
> but may actually cause serious trouble.
>
> There are other issues (e.g. the row count estimate of the CTE is
> seriously off, most likely because of the HashAggregate in the outer
> branch), but that's a secondary issue IMHO.
>
> Vivekanand, try this (in the order of intrusiveness):
>
> (1) Get rid of the CTE, and just replace it with subselect in the FROM
> part of the query, so instead of this:
>
> WITH valid_executions AS (...)
> SELECT ... FROM ... JOIN valid_executions ON (...)
>
> you'll have something like this:
>
> SELECT ... FROM ... JOIN (...) AS valid_executions ON (...)
>
> This way the subselect will optimized properly.
>
>
> (2) Replace the CTE with a materialized view, or a temporary table.
> This has both advantages and disadvantages - the main advantage is
> that you can create indexes, collect statistics. Disadvantage is
> you have to refresh the MV, fill temporary table etc.
>
> I expect (1) to improve the performance significantly, and (2) might
> improve it even further by fixing the misestimates.
>
>
> regards
>
> --
> Tomas Vondra http://www.2ndQuadrant.com/
> PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
>
>
> --
> Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-performance
>
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Performance issues
2015-03-13 11:44 Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 12:58 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 13:03 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 13:10 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 19:59 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 20:36 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-13 20:46 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 22:03 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 22:25 ` Re: Performance issues Scott Marlowe <scott.marlowe@gmail.com>
2015-03-13 23:28 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-14 00:12 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
@ 2015-03-16 19:43 ` Jim Nasby <Jim.Nasby@BlueTreble.com>
2015-03-16 20:59 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
1 sibling, 1 reply; 59+ messages in thread
From: Jim Nasby @ 2015-03-16 19:43 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; vjoshi@zetainteractive.com; Scott Marlowe <scott.marlowe@gmail.com>; Varadharajan Mukundan <srinathsmn@gmail.com>; +Cc: pgsql-performance
On 3/13/15 7:12 PM, Tomas Vondra wrote:
> (4) I suspect many of the relations referenced in the views are not
> actually needed in the query, i.e. the join is performed but
> then it's just discarded because those columns are not used.
> Try to simplify the views as much has possible - remove all the
> tables that are not really necessary to run the query. If two
> queries need different tables, maybe defining two views is
> a better approach.
A better alternative with multi-purpose views is to use an outer join
instead of an inner join. With an outer join if you ultimately don't
refer to any of the columns in a particular table Postgres will remove
the table from the query completely.
--
Jim Nasby, Data Architect, Blue Treble Consulting
Data in Trouble? Get it in Treble! http://BlueTreble.com
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Performance issues
2015-03-13 11:44 Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 12:58 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 13:03 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 13:10 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 19:59 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 20:36 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-13 20:46 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 22:03 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 22:25 ` Re: Performance issues Scott Marlowe <scott.marlowe@gmail.com>
2015-03-13 23:28 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-14 00:12 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-16 19:43 ` Re: Performance issues Jim Nasby <Jim.Nasby@BlueTreble.com>
@ 2015-03-16 20:59 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 00:06 ` Re: Performance issues Jim Nasby <Jim.Nasby@BlueTreble.com>
0 siblings, 1 reply; 59+ messages in thread
From: Tomas Vondra @ 2015-03-16 20:59 UTC (permalink / raw)
To: Jim Nasby <Jim.Nasby@BlueTreble.com>; vjoshi@zetainteractive.com, Scott Marlowe <scott.marlowe@gmail.com>; Varadharajan Mukundan <srinathsmn@gmail.com>; +Cc: pgsql-performance
On 16.3.2015 20:43, Jim Nasby wrote:
> On 3/13/15 7:12 PM, Tomas Vondra wrote:
>> (4) I suspect many of the relations referenced in the views are not
>> actually needed in the query, i.e. the join is performed but
>> then it's just discarded because those columns are not used.
>> Try to simplify the views as much has possible - remove all the
>> tables that are not really necessary to run the query. If two
>> queries need different tables, maybe defining two views is
>> a better approach.
>
> A better alternative with multi-purpose views is to use an outer
> join instead of an inner join. With an outer join if you ultimately
> don't refer to any of the columns in a particular table Postgres will
> remove the table from the query completely.
Really? Because a quick test suggests otherwise:
db=# create table test_a (id int);
CREATE TABLE
db=# create table test_b (id int);
CREATE TABLE
db=# explain select test_a.* from test_a left join test_b using (id);
QUERY PLAN
----------------------------------------------------------------------
Merge Left Join (cost=359.57..860.00 rows=32512 width=4)
Merge Cond: (test_a.id = test_b.id)
-> Sort (cost=179.78..186.16 rows=2550 width=4)
Sort Key: test_a.id
-> Seq Scan on test_a (cost=0.00..35.50 rows=2550 width=4)
-> Sort (cost=179.78..186.16 rows=2550 width=4)
Sort Key: test_b.id
-> Seq Scan on test_b (cost=0.00..35.50 rows=2550 width=4)
(8 rows)
Also, how would that work with duplicate rows in the referenced table?
--
Tomas Vondra http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Performance issues
2015-03-13 11:44 Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 12:58 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 13:03 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 13:10 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 19:59 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 20:36 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-13 20:46 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 22:03 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 22:25 ` Re: Performance issues Scott Marlowe <scott.marlowe@gmail.com>
2015-03-13 23:28 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-14 00:12 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-16 19:43 ` Re: Performance issues Jim Nasby <Jim.Nasby@BlueTreble.com>
2015-03-16 20:59 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
@ 2015-03-17 00:06 ` Jim Nasby <Jim.Nasby@BlueTreble.com>
2015-03-17 07:41 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-17 11:07 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
0 siblings, 2 replies; 59+ messages in thread
From: Jim Nasby @ 2015-03-17 00:06 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; vjoshi@zetainteractive.com; Scott Marlowe <scott.marlowe@gmail.com>; Varadharajan Mukundan <srinathsmn@gmail.com>; +Cc: pgsql-performance
On 3/16/15 3:59 PM, Tomas Vondra wrote:
> On 16.3.2015 20:43, Jim Nasby wrote:
>> On 3/13/15 7:12 PM, Tomas Vondra wrote:
>>> (4) I suspect many of the relations referenced in the views are not
>>> actually needed in the query, i.e. the join is performed but
>>> then it's just discarded because those columns are not used.
>>> Try to simplify the views as much has possible - remove all the
>>> tables that are not really necessary to run the query. If two
>>> queries need different tables, maybe defining two views is
>>> a better approach.
>>
>> A better alternative with multi-purpose views is to use an outer
>> join instead of an inner join. With an outer join if you ultimately
>> don't refer to any of the columns in a particular table Postgres will
>> remove the table from the query completely.
>
> Really? Because a quick test suggests otherwise:
>
> db=# create table test_a (id int);
> CREATE TABLE
> db=# create table test_b (id int);
> CREATE TABLE
> db=# explain select test_a.* from test_a left join test_b using (id);
> QUERY PLAN
> ----------------------------------------------------------------------
> Merge Left Join (cost=359.57..860.00 rows=32512 width=4)
> Merge Cond: (test_a.id = test_b.id)
> -> Sort (cost=179.78..186.16 rows=2550 width=4)
> Sort Key: test_a.id
> -> Seq Scan on test_a (cost=0.00..35.50 rows=2550 width=4)
> -> Sort (cost=179.78..186.16 rows=2550 width=4)
> Sort Key: test_b.id
> -> Seq Scan on test_b (cost=0.00..35.50 rows=2550 width=4)
> (8 rows)
>
> Also, how would that work with duplicate rows in the referenced table?
Right, I neglected to mention that the omitted table must also be unique
on the join key:
decibel@decina.attlocal=# create table a(a_id serial primary key);
CREATE TABLE
decibel@decina.attlocal=# create table b(a_id int);
CREATE TABLE
decibel@decina.attlocal=# explain analyze select a.* from a left join b
using(a_id);
QUERY PLAN
-----------------------------------------------------------------------------------------------------------
Hash Right Join (cost=67.38..137.94 rows=2550 width=4) (actual
time=0.035..0.035 rows=0 loops=1)
Hash Cond: (b.a_id = a.a_id)
-> Seq Scan on b (cost=0.00..35.50 rows=2550 width=4) (never executed)
-> Hash (cost=35.50..35.50 rows=2550 width=4) (actual
time=0.002..0.002 rows=0 loops=1)
Buckets: 4096 Batches: 1 Memory Usage: 32kB
-> Seq Scan on a (cost=0.00..35.50 rows=2550 width=4)
(actual time=0.001..0.001 rows=0 loops=1)
Planning time: 0.380 ms
Execution time: 0.086 ms
(8 rows)
decibel@decina.attlocal=# alter table b add primary key(a_id);
ALTER TABLE
decibel@decina.attlocal=# explain analyze select a.* from a left join b
using(a_id);
QUERY PLAN
-----------------------------------------------------------------------------------------------
Seq Scan on a (cost=0.00..35.50 rows=2550 width=4) (actual
time=0.001..0.001 rows=0 loops=1)
Planning time: 0.247 ms
Execution time: 0.029 ms
(3 rows)
decibel@decina.attlocal=# alter table a drop constraint a_pkey;
ALTER TABLE
decibel@decina.attlocal=# explain analyze select a.* from a left join b
using(a_id);
QUERY PLAN
-----------------------------------------------------------------------------------------------
Seq Scan on a (cost=0.00..35.50 rows=2550 width=4) (actual
time=0.001..0.001 rows=0 loops=1)
Planning time: 0.098 ms
Execution time: 0.011 ms
(3 rows)
--
Jim Nasby, Data Architect, Blue Treble Consulting
Data in Trouble? Get it in Treble! http://BlueTreble.com
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Performance issues
2015-03-13 11:44 Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 12:58 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 13:03 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 13:10 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 19:59 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 20:36 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-13 20:46 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 22:03 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 22:25 ` Re: Performance issues Scott Marlowe <scott.marlowe@gmail.com>
2015-03-13 23:28 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-14 00:12 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-16 19:43 ` Re: Performance issues Jim Nasby <Jim.Nasby@BlueTreble.com>
2015-03-16 20:59 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 00:06 ` Re: Performance issues Jim Nasby <Jim.Nasby@BlueTreble.com>
@ 2015-03-17 07:41 ` Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-17 11:42 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
1 sibling, 1 reply; 59+ messages in thread
From: Vivekanand Joshi @ 2015-03-17 07:41 UTC (permalink / raw)
To: Jim Nasby <Jim.Nasby@bluetreble.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>; Varadharajan Mukundan <srinathsmn@gmail.com>; +Cc: pgsql-performance
Hi Guys,
Next level of query is following:
If this works, I guess 90% of the problem will be solved.
SELECT
COUNT(DISTINCT TARGET_ID)
FROM
S_V_F_PROMOTION_HISTORY_EMAIL PH
INNER JOIN S_V_D_CAMPAIGN_HIERARCHY CH
ON PH.TOUCHPOINT_EXECUTION_ID =
CH.TOUCHPOINT_EXECUTION_ID
WHERE
1=1
AND SEND_DT >= '2014-03-13'
AND SEND_DT <= '2015-03-14'
In this query, I am joining two views which were made earlier with CTEs. I
have replaced the CTE's with subqueries. The view were giving me output in
around 5-10 minutes and now I am getting the same result in around 3-4
seconds.
But when I executed the query written above, I am again stuck. I am
attaching the query plan as well the link.
http://explain.depesz.com/s/REeu
I can see most of the time is spending inside a nested loop and total
costs comes out be cost=338203.81..338203.82.
How to take care of this? I need to run this query in a report so I cannot
create a table like select * from views and then join the table. If I do
that I am getting the answer of whole big query in some 6-7 seconds. But
that is not feasible. A report (Jasper can have only one single (big/small
query).
Let me know if you need any other information.
Thanks a ton!
Vivek
-----Original Message-----
From: Jim Nasby [mailto:Jim.Nasby@BlueTreble.com]
Sent: Tuesday, March 17, 2015 5:36 AM
To: Tomas Vondra; vjoshi@zetainteractive.com; Scott Marlowe; Varadharajan
Mukundan
Cc: pgsql-performance@postgresql.org
Subject: Re: [PERFORM] Performance issues
On 3/16/15 3:59 PM, Tomas Vondra wrote:
> On 16.3.2015 20:43, Jim Nasby wrote:
>> On 3/13/15 7:12 PM, Tomas Vondra wrote:
>>> (4) I suspect many of the relations referenced in the views are not
>>> actually needed in the query, i.e. the join is performed but
>>> then it's just discarded because those columns are not used.
>>> Try to simplify the views as much has possible - remove all the
>>> tables that are not really necessary to run the query. If two
>>> queries need different tables, maybe defining two views is
>>> a better approach.
>>
>> A better alternative with multi-purpose views is to use an outer join
>> instead of an inner join. With an outer join if you ultimately don't
>> refer to any of the columns in a particular table Postgres will
>> remove the table from the query completely.
>
> Really? Because a quick test suggests otherwise:
>
> db=# create table test_a (id int);
> CREATE TABLE
> db=# create table test_b (id int);
> CREATE TABLE
> db=# explain select test_a.* from test_a left join test_b using (id);
> QUERY PLAN
> ----------------------------------------------------------------------
> Merge Left Join (cost=359.57..860.00 rows=32512 width=4)
> Merge Cond: (test_a.id = test_b.id)
> -> Sort (cost=179.78..186.16 rows=2550 width=4)
> Sort Key: test_a.id
> -> Seq Scan on test_a (cost=0.00..35.50 rows=2550 width=4)
> -> Sort (cost=179.78..186.16 rows=2550 width=4)
> Sort Key: test_b.id
> -> Seq Scan on test_b (cost=0.00..35.50 rows=2550 width=4)
> (8 rows)
>
> Also, how would that work with duplicate rows in the referenced table?
Right, I neglected to mention that the omitted table must also be unique
on the join key:
decibel@decina.attlocal=# create table a(a_id serial primary key); CREATE
TABLE decibel@decina.attlocal=# create table b(a_id int); CREATE TABLE
decibel@decina.attlocal=# explain analyze select a.* from a left join b
using(a_id);
QUERY PLAN
--------------------------------------------------------------------------
---------------------------------
Hash Right Join (cost=67.38..137.94 rows=2550 width=4) (actual
time=0.035..0.035 rows=0 loops=1)
Hash Cond: (b.a_id = a.a_id)
-> Seq Scan on b (cost=0.00..35.50 rows=2550 width=4) (never
executed)
-> Hash (cost=35.50..35.50 rows=2550 width=4) (actual
time=0.002..0.002 rows=0 loops=1)
Buckets: 4096 Batches: 1 Memory Usage: 32kB
-> Seq Scan on a (cost=0.00..35.50 rows=2550 width=4) (actual
time=0.001..0.001 rows=0 loops=1)
Planning time: 0.380 ms
Execution time: 0.086 ms
(8 rows)
decibel@decina.attlocal=# alter table b add primary key(a_id); ALTER TABLE
decibel@decina.attlocal=# explain analyze select a.* from a left join b
using(a_id);
QUERY PLAN
--------------------------------------------------------------------------
---------------------
Seq Scan on a (cost=0.00..35.50 rows=2550 width=4) (actual
time=0.001..0.001 rows=0 loops=1)
Planning time: 0.247 ms
Execution time: 0.029 ms
(3 rows)
decibel@decina.attlocal=# alter table a drop constraint a_pkey; ALTER
TABLE decibel@decina.attlocal=# explain analyze select a.* from a left
join b using(a_id);
QUERY PLAN
--------------------------------------------------------------------------
---------------------
Seq Scan on a (cost=0.00..35.50 rows=2550 width=4) (actual
time=0.001..0.001 rows=0 loops=1)
Planning time: 0.098 ms
Execution time: 0.011 ms
(3 rows)
--
Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it
in Treble! http://BlueTreble.com
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
Attachments:
Explain.sql (text/plain, 7.4K) download
Aggregate (cost=338203.81..338203.82 rows=1 width=8)
-> Nested Loop (cost=51266.84..338203.81 rows=1 width=8)
Join Filter: (s_f_touchpoint_execution_status_history_1.touchpoint_execution_id = s_f_touchpoint_execution_status_history.touchpoint_execution_id)
-> Nested Loop (cost=32782.19..317837.83 rows=1 width=32)
-> Nested Loop (cost=32781.90..317837.50 rows=1 width=40)
-> Nested Loop (cost=32781.61..317829.18 rows=1 width=24)
Join Filter: (base.touchpoint_execution_id = s_f_touchpoint_execution_status_history_1.touchpoint_execution_id)
-> Unique (cost=32781.61..34495.50 rows=1 width=8)
-> Merge Join (cost=32781.61..34495.50 rows=1 width=8)
Merge Cond: ((s_f_touchpoint_execution_status_history_1.touchpoint_execution_id = s_f_touchpoint_execution_status_history_1_1.tou
chpoint_execution_id) AND (s_f_touchpoint_execution_status_history_1.creation_dt = (max(s_f_touchpoint_execution_status_history_1_1.creation_dt))))
-> Sort (cost=19697.87..20098.14 rows=160107 width=16)
Sort Key: s_f_touchpoint_execution_status_history_1.touchpoint_execution_id, s_f_touchpoint_execution_status_history_1.crea
tion_dt
-> Seq Scan on s_f_touchpoint_execution_status_history s_f_touchpoint_execution_status_history_1 (cost=0.00..5857.68 rows
=160107 width=16)
Filter: (touchpoint_execution_status_type_id = ANY ('{3,4,6}'::integer[]))
-> Sort (cost=13083.74..13254.76 rows=68410 width=16)
Sort Key: s_f_touchpoint_execution_status_history_1_1.touchpoint_execution_id, (max(s_f_touchpoint_execution_status_history
_1_1.creation_dt))
-> HashAggregate (cost=6221.56..6905.66 rows=68410 width=16)
-> Seq Scan on s_f_touchpoint_execution_status_history s_f_touchpoint_execution_status_history_1_1 (cost=0.00..4766
.04 rows=291104 width=16)
-> Seq Scan on s_f_promotion_history base (cost=0.00..283333.66 rows=1 width=32)
Filter: ((send_dt >= '2014-03-13 00:00:00'::timestamp without time zone) AND (send_dt <= '2015-03-14 00:00:00'::timestamp without time
zone))
-> Index Scan using idx_s_d_touchpoint_execution_id on s_d_touchpoint_execution tpe (cost=0.29..8.31 rows=1 width=16)
Index Cond: (touchpoint_execution_id = s_f_touchpoint_execution_status_history_1.touchpoint_execution_id)
-> Index Only Scan using s_d_touchpoint_pkey on s_d_touchpoint tp (cost=0.29..0.32 rows=1 width=8)
Index Cond: ((touchpoint_id = tpe.touchpoint_id) AND (channel_type_id = 1))
-> Nested Loop (cost=18484.65..20365.97 rows=1 width=24)
-> Hash Join (cost=18484.52..20365.80 rows=1 width=26)
Hash Cond: ((s_f_touchpoint_execution_status_history_1_2.touchpoint_execution_id = s_f_touchpoint_execution_status_history.touchpoint_execution_id)
AND ((max(s_f_touchpoint_execution_status_history_1_2.creation_dt)) = s_f_touchpoint_execution_status_history.creation_dt))
-> HashAggregate (cost=6221.56..6905.66 rows=68410 width=16)
-> Seq Scan on s_f_touchpoint_execution_status_history s_f_touchpoint_execution_status_history_1_2 (cost=0.00..4766.04 rows=291104 width=16
)
-> Hash (cost=12262.93..12262.93 rows=2 width=26)
-> Hash Join (cost=6259.57..12262.93 rows=2 width=26)
Hash Cond: (s_f_touchpoint_execution_status_history.touchpoint_execution_id = tp_exec.touchpoint_execution_id)
-> Seq Scan on s_f_touchpoint_execution_status_history (cost=0.00..5493.80 rows=135878 width=16)
Filter: (touchpoint_execution_status_type_id = ANY ('{3,4}'::integer[]))
-> Hash (cost=6259.55..6259.55 rows=1 width=10)
-> Nested Loop Left Join (cost=1955.27..6259.55 rows=1 width=10)
-> Nested Loop (cost=1954.99..6259.24 rows=1 width=18)
-> Nested Loop (cost=1954.71..6258.92 rows=1 width=34)
Join Filter: (camp_exec.campaign_id = wave.campaign_id)
-> Nested Loop (cost=1954.42..6254.67 rows=13 width=42)
-> Hash Join (cost=1954.13..6249.67 rows=13 width=42)
Hash Cond: ((tp_exec.touchpoint_id = tp_1.touchpoint_id) AND (wave_exec.wave_id = tp_1.wave_id))
-> Hash Join (cost=1576.83..4595.51 rows=72956 width=40)
Hash Cond: (tp_exec.wave_execution_id = wave_exec.wave_execution_id)
-> Seq Scan on s_d_touchpoint_execution tp_exec (cost=0.00..1559.56 rows=72956 width=34)
-> Hash (cost=1001.37..1001.37 rows=46037 width=24)
-> Seq Scan on s_d_wave_execution wave_exec (cost=0.00..1001.37 rows=46037 width=24)
-> Hash (cost=212.72..212.72 rows=10972 width=18)
-> Seq Scan on s_d_touchpoint tp_1 (cost=0.00..212.72 rows=10972 width=18)
-> Index Scan using s_d_campaign_execution_idx on s_d_campaign_execution camp_exec (cost=0.29..0.37 row
s=1 width=18)
Index Cond: (campaign_execution_id = wave_exec.campaign_execution_id)
-> Index Scan using s_d_wave_pkey on s_d_wave wave (cost=0.29..0.31 rows=1 width=16)
Index Cond: (wave_id = wave_exec.wave_id)
-> Index Scan using s_d_campaign_pkey on s_d_campaign camp (cost=0.29..0.32 rows=1 width=10)
Index Cond: (campaign_id = camp_exec.campaign_id)
-> Index Only Scan using s_d_content_pkey on s_d_content content (cost=0.28..0.30 rows=1 width=8)
Index Cond: (content_id = tp_exec.content_id)
-> Index Only Scan using d_channel_pk on s_d_channel_type channel (cost=0.13..0.15 rows=1 width=2)
Index Cond: (channel_type_id = tp_1.channel_type_id)
* Re: Performance issues
2015-03-13 11:44 Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 12:58 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 13:03 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 13:10 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 19:59 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 20:36 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-13 20:46 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 22:03 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 22:25 ` Re: Performance issues Scott Marlowe <scott.marlowe@gmail.com>
2015-03-13 23:28 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-14 00:12 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-16 19:43 ` Re: Performance issues Jim Nasby <Jim.Nasby@BlueTreble.com>
2015-03-16 20:59 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 00:06 ` Re: Performance issues Jim Nasby <Jim.Nasby@BlueTreble.com>
2015-03-17 07:41 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
@ 2015-03-17 11:42 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 0 replies; 59+ messages in thread
From: Tomas Vondra @ 2015-03-17 11:42 UTC (permalink / raw)
To: vjoshi@zetainteractive.com, Jim Nasby <Jim.Nasby@bluetreble.com>; Scott Marlowe <scott.marlowe@gmail.com>; Varadharajan Mukundan <srinathsmn@gmail.com>; +Cc: pgsql-performance
Hi,
On 17.3.2015 08:41, Vivekanand Joshi wrote:
> Hi Guys,
>
> Next level of query is following:
>
> If this works, I guess 90% of the problem will be solved.
>
> SELECT
> COUNT(DISTINCT TARGET_ID)
> FROM
> S_V_F_PROMOTION_HISTORY_EMAIL PH
> INNER JOIN S_V_D_CAMPAIGN_HIERARCHY CH
> ON PH.TOUCHPOINT_EXECUTION_ID =
> CH.TOUCHPOINT_EXECUTION_ID
> WHERE
> 1=1
> AND SEND_DT >= '2014-03-13'
> AND SEND_DT <= '2015-03-14'
>
>
> In this query, I am joining two views which were made earlier with CTEs. I
> have replaced the CTE's with subqueries. The view were giving me output in
> around 5-10 minutes and now I am getting the same result in around 3-4
> seconds.
>
> But when I executed the query written above, I am again stuck. I am
> attaching the query plan as well the link.
>
> http://explain.depesz.com/s/REeu
>
> I can see most of the time is spending inside a nested loop and total
> costs comes out be cost=338203.81..338203.82.
Most of that cost comes from this:
Seq Scan on s_f_promotion_history base (cost=0.00..283,333.66 rows=1
width=32)
Filter: ((send_dt >= '2014-03-13 00:00:00'::timestamp without time
zone) AND (send_dt <= '2015-03-14 00:00:00'::timestamp without time
That's a bit weird, I guess. If you analyze this part of the query
separately, i.e.
EXPLAIN ANALYZE SELECT * FROM s_f_promotion_history
WHERE (send_dt >= '2014-03-13 00:00:00')
AND (send_dt <= '2015-03-14 00:00:00')
what do you get?
I suspect it's used in EXISTS, i.e. something like this:
... WHERE EXISTS (SELECT * FROM s_f_promotion_history
WHERE ... send_dt conditions ...
AND touchpoint_execution_id =
s_f_touchpoint_execution_status_history_1.touchpoint_execution_id)
and this is transformed into a nested loop join. If there's a
misestimate, this may be quite expensive - try to create index on
s_f_promotion_history (touchpoint_execution_id, send_date)
regards
--
Tomas Vondra http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Performance issues
2015-03-13 11:44 Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 12:58 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 13:03 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 13:10 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 19:59 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 20:36 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-13 20:46 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 22:03 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 22:25 ` Re: Performance issues Scott Marlowe <scott.marlowe@gmail.com>
2015-03-13 23:28 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-14 00:12 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-16 19:43 ` Re: Performance issues Jim Nasby <Jim.Nasby@BlueTreble.com>
2015-03-16 20:59 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 00:06 ` Re: Performance issues Jim Nasby <Jim.Nasby@BlueTreble.com>
@ 2015-03-17 11:07 ` Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-17 11:45 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
1 sibling, 1 reply; 59+ messages in thread
From: Vivekanand Joshi @ 2015-03-17 11:07 UTC (permalink / raw)
To: vjoshi@zetainteractive.com, Jim Nasby <Jim.Nasby@bluetreble.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Scott Marlowe <scott.marlowe@gmail.com>; Varadharajan Mukundan <srinathsmn@gmail.com>; +Cc: pgsql-performance
EXPLAIN ANALYZE didn't give result even after three hours.
-----Original Message-----
From: Vivekanand Joshi [mailto:vjoshi@zetainteractive.com]
Sent: Tuesday, March 17, 2015 1:11 PM
To: 'Jim Nasby'; 'Tomas Vondra'; 'Scott Marlowe'; 'Varadharajan Mukundan'
Cc: 'pgsql-performance@postgresql.org'
Subject: RE: [PERFORM] Performance issues
Hi Guys,
Next level of query is following:
If this works, I guess 90% of the problem will be solved.
SELECT
COUNT(DISTINCT TARGET_ID)
FROM
S_V_F_PROMOTION_HISTORY_EMAIL PH
INNER JOIN S_V_D_CAMPAIGN_HIERARCHY CH
ON PH.TOUCHPOINT_EXECUTION_ID =
CH.TOUCHPOINT_EXECUTION_ID
WHERE
1=1
AND SEND_DT >= '2014-03-13'
AND SEND_DT <= '2015-03-14'
In this query, I am joining two views which were made earlier with CTEs. I
have replaced the CTE's with subqueries. The view were giving me output in
around 5-10 minutes and now I am getting the same result in around 3-4
seconds.
But when I executed the query written above, I am again stuck. I am
attaching the query plan as well the link.
http://explain.depesz.com/s/REeu
I can see most of the time is spending inside a nested loop and total
costs comes out be cost=338203.81..338203.82.
How to take care of this? I need to run this query in a report so I cannot
create a table like select * from views and then join the table. If I do
that I am getting the answer of whole big query in some 6-7 seconds. But
that is not feasible. A report (Jasper can have only one single (big/small
query).
Let me know if you need any other information.
Thanks a ton!
Vivek
-----Original Message-----
From: Jim Nasby [mailto:Jim.Nasby@BlueTreble.com]
Sent: Tuesday, March 17, 2015 5:36 AM
To: Tomas Vondra; vjoshi@zetainteractive.com; Scott Marlowe; Varadharajan
Mukundan
Cc: pgsql-performance@postgresql.org
Subject: Re: [PERFORM] Performance issues
On 3/16/15 3:59 PM, Tomas Vondra wrote:
> On 16.3.2015 20:43, Jim Nasby wrote:
>> On 3/13/15 7:12 PM, Tomas Vondra wrote:
>>> (4) I suspect many of the relations referenced in the views are not
>>> actually needed in the query, i.e. the join is performed but
>>> then it's just discarded because those columns are not used.
>>> Try to simplify the views as much has possible - remove all the
>>> tables that are not really necessary to run the query. If two
>>> queries need different tables, maybe defining two views is
>>> a better approach.
>>
>> A better alternative with multi-purpose views is to use an outer join
>> instead of an inner join. With an outer join if you ultimately don't
>> refer to any of the columns in a particular table Postgres will
>> remove the table from the query completely.
>
> Really? Because a quick test suggests otherwise:
>
> db=# create table test_a (id int);
> CREATE TABLE
> db=# create table test_b (id int);
> CREATE TABLE
> db=# explain select test_a.* from test_a left join test_b using (id);
> QUERY PLAN
> ----------------------------------------------------------------------
> Merge Left Join (cost=359.57..860.00 rows=32512 width=4)
> Merge Cond: (test_a.id = test_b.id)
> -> Sort (cost=179.78..186.16 rows=2550 width=4)
> Sort Key: test_a.id
> -> Seq Scan on test_a (cost=0.00..35.50 rows=2550 width=4)
> -> Sort (cost=179.78..186.16 rows=2550 width=4)
> Sort Key: test_b.id
> -> Seq Scan on test_b (cost=0.00..35.50 rows=2550 width=4)
> (8 rows)
>
> Also, how would that work with duplicate rows in the referenced table?
Right, I neglected to mention that the omitted table must also be unique
on the join key:
decibel@decina.attlocal=# create table a(a_id serial primary key); CREATE
TABLE decibel@decina.attlocal=# create table b(a_id int); CREATE TABLE
decibel@decina.attlocal=# explain analyze select a.* from a left join b
using(a_id);
QUERY PLAN
--------------------------------------------------------------------------
---------------------------------
Hash Right Join (cost=67.38..137.94 rows=2550 width=4) (actual
time=0.035..0.035 rows=0 loops=1)
Hash Cond: (b.a_id = a.a_id)
-> Seq Scan on b (cost=0.00..35.50 rows=2550 width=4) (never
executed)
-> Hash (cost=35.50..35.50 rows=2550 width=4) (actual
time=0.002..0.002 rows=0 loops=1)
Buckets: 4096 Batches: 1 Memory Usage: 32kB
-> Seq Scan on a (cost=0.00..35.50 rows=2550 width=4) (actual
time=0.001..0.001 rows=0 loops=1)
Planning time: 0.380 ms
Execution time: 0.086 ms
(8 rows)
decibel@decina.attlocal=# alter table b add primary key(a_id); ALTER TABLE
decibel@decina.attlocal=# explain analyze select a.* from a left join b
using(a_id);
QUERY PLAN
--------------------------------------------------------------------------
---------------------
Seq Scan on a (cost=0.00..35.50 rows=2550 width=4) (actual
time=0.001..0.001 rows=0 loops=1)
Planning time: 0.247 ms
Execution time: 0.029 ms
(3 rows)
decibel@decina.attlocal=# alter table a drop constraint a_pkey; ALTER
TABLE decibel@decina.attlocal=# explain analyze select a.* from a left
join b using(a_id);
QUERY PLAN
--------------------------------------------------------------------------
---------------------
Seq Scan on a (cost=0.00..35.50 rows=2550 width=4) (actual
time=0.001..0.001 rows=0 loops=1)
Planning time: 0.098 ms
Execution time: 0.011 ms
(3 rows)
--
Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it
in Treble! http://BlueTreble.com
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Performance issues
2015-03-13 11:44 Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 12:58 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 13:03 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 13:10 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 19:59 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 20:36 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-13 20:46 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 22:03 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 22:25 ` Re: Performance issues Scott Marlowe <scott.marlowe@gmail.com>
2015-03-13 23:28 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-14 00:12 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-16 19:43 ` Re: Performance issues Jim Nasby <Jim.Nasby@BlueTreble.com>
2015-03-16 20:59 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 00:06 ` Re: Performance issues Jim Nasby <Jim.Nasby@BlueTreble.com>
2015-03-17 11:07 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
@ 2015-03-17 11:45 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 12:05 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-17 12:10 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
0 siblings, 2 replies; 59+ messages in thread
From: Tomas Vondra @ 2015-03-17 11:45 UTC (permalink / raw)
To: vjoshi@zetainteractive.com, Jim Nasby <Jim.Nasby@bluetreble.com>; Scott Marlowe <scott.marlowe@gmail.com>; Varadharajan Mukundan <srinathsmn@gmail.com>; +Cc: pgsql-performance
On 17.3.2015 12:07, Vivekanand Joshi wrote:
> EXPLAIN ANALYZE didn't give result even after three hours.
In that case the only thing you can do is 'slice' the query into smaller
parts (representing subtrees of the plan), and analyze those first. Look
for misestimates (significant differences between estimated and actual
row counts, and very expensive parts).
We can't do that, because we don't have your data or queries, and
without the explain analyze it's difficult to give advices.
--
Tomas Vondra http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Performance issues
2015-03-13 11:44 Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 12:58 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 13:03 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 13:10 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 19:59 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 20:36 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-13 20:46 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 22:03 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 22:25 ` Re: Performance issues Scott Marlowe <scott.marlowe@gmail.com>
2015-03-13 23:28 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-14 00:12 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-16 19:43 ` Re: Performance issues Jim Nasby <Jim.Nasby@BlueTreble.com>
2015-03-16 20:59 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 00:06 ` Re: Performance issues Jim Nasby <Jim.Nasby@BlueTreble.com>
2015-03-17 11:07 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-17 11:45 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
@ 2015-03-17 12:05 ` Vivekanand Joshi <vjoshi@zetainteractive.com>
1 sibling, 0 replies; 59+ messages in thread
From: Vivekanand Joshi @ 2015-03-17 12:05 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Jim Nasby <Jim.Nasby@bluetreble.com>; Scott Marlowe <scott.marlowe@gmail.com>; Varadharajan Mukundan <srinathsmn@gmail.com>; +Cc: pgsql-performance
Hi Tomas,
This is what I am getting,
EXPLAIN ANALYZE SELECT * FROM s_f_promotion_history WHERE (send_dt >=
'2014-03-13 00:00:00');
QUERY
PLAN
-------------------------------------------------------------------------------------------------------------------------------------------------
Seq Scan on s_f_promotion_history (cost=0.00..283333.66 rows=1 width=74)
(actual time=711.023..1136.393 rows=1338 loops=1)
Filter: ((send_dt >= '2014-03-13 00:00:00'::timestamp without time zone)
AND (send_dt <= '2015-03-14 00:00:00'::timestamp without time zone))
Rows Removed by Filter: 9998662
Total runtime: 1170.682 ms
CREATE INDEX idx_pr_history ON
S_F_PROMOTION_HISTORY(touchpoint_execution_id, send_dt);
After Creating Index:
QUERY
PLAN
-----------------------------------------------------------------------------------------------------------------------------------------------------
Index Scan using idx_pr_history on s_f_promotion_history
(cost=0.43..254028.45 rows=1 width=74) (actual time=375.796..604.587
rows=1338 loops=1)
Index Cond: ((send_dt >= '2014-03-13 00:00:00'::timestamp without time
zone) AND (send_dt <= '2015-03-14 00:00:00'::timestamp without time zone))
Total runtime: 604.733 ms
The query I gave you is the smallest query, it is using two views and both
the views I have changed by using subqueries instead of CTEs. When I join
these two views, it is not getting completed at all.
Explain analyze plan for view s_v_f_promotion_history_email:
http://explain.depesz.com/s/ure
Explain analyze plan for view s_v_d_campaign_hierarchy :
http://explain.depesz.com/s/WxI
Regards,
Vivek
-----Original Message-----
From: Tomas Vondra [mailto:tomas.vondra@2ndquadrant.com]
Sent: Tuesday, March 17, 2015 5:15 PM
To: vjoshi@zetainteractive.com; Jim Nasby; Scott Marlowe; Varadharajan
Mukundan
Cc: pgsql-performance@postgresql.org
Subject: Re: [PERFORM] Performance issues
On 17.3.2015 12:07, Vivekanand Joshi wrote:
> EXPLAIN ANALYZE didn't give result even after three hours.
In that case the only thing you can do is 'slice' the query into smaller
parts (representing subtrees of the plan), and analyze those first. Look
for misestimates (significant differences between estimated and actual
row counts, and very expensive parts).
We can't do that, because we don't have your data or queries, and
without the explain analyze it's difficult to give advices.
--
Tomas Vondra http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Performance issues
2015-03-13 11:44 Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 12:58 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 13:03 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 13:10 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 19:59 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 20:36 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-13 20:46 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 22:03 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 22:25 ` Re: Performance issues Scott Marlowe <scott.marlowe@gmail.com>
2015-03-13 23:28 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-14 00:12 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-16 19:43 ` Re: Performance issues Jim Nasby <Jim.Nasby@BlueTreble.com>
2015-03-16 20:59 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 00:06 ` Re: Performance issues Jim Nasby <Jim.Nasby@BlueTreble.com>
2015-03-17 11:07 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-17 11:45 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
@ 2015-03-17 12:10 ` Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-17 13:55 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
1 sibling, 1 reply; 59+ messages in thread
From: Vivekanand Joshi @ 2015-03-17 12:10 UTC (permalink / raw)
To: vjoshi@zetainteractive.com, Tomas Vondra <tomas.vondra@2ndquadrant.com>; Jim Nasby <Jim.Nasby@bluetreble.com>; Scott Marlowe <scott.marlowe@gmail.com>; Varadharajan Mukundan <srinathsmn@gmail.com>; +Cc: pgsql-performance
Attaching explain analyze file as well.
Vivek
-----Original Message-----
From: Vivekanand Joshi [mailto:vjoshi@zetainteractive.com]
Sent: Tuesday, March 17, 2015 5:36 PM
To: 'Tomas Vondra'; 'Jim Nasby'; 'Scott Marlowe'; 'Varadharajan Mukundan'
Cc: 'pgsql-performance@postgresql.org'
Subject: RE: [PERFORM] Performance issues
Hi Tomas,
This is what I am getting,
EXPLAIN ANALYZE SELECT * FROM s_f_promotion_history WHERE (send_dt >=
'2014-03-13 00:00:00');
QUERY
PLAN
-------------------------------------------------------------------------------------------------------------------------------------------------
Seq Scan on s_f_promotion_history (cost=0.00..283333.66 rows=1 width=74)
(actual time=711.023..1136.393 rows=1338 loops=1)
Filter: ((send_dt >= '2014-03-13 00:00:00'::timestamp without time zone)
AND (send_dt <= '2015-03-14 00:00:00'::timestamp without time zone))
Rows Removed by Filter: 9998662
Total runtime: 1170.682 ms
CREATE INDEX idx_pr_history ON
S_F_PROMOTION_HISTORY(touchpoint_execution_id, send_dt);
After Creating Index:
QUERY
PLAN
-----------------------------------------------------------------------------------------------------------------------------------------------------
Index Scan using idx_pr_history on s_f_promotion_history
(cost=0.43..254028.45 rows=1 width=74) (actual time=375.796..604.587
rows=1338 loops=1)
Index Cond: ((send_dt >= '2014-03-13 00:00:00'::timestamp without time
zone) AND (send_dt <= '2015-03-14 00:00:00'::timestamp without time zone))
Total runtime: 604.733 ms
The query I gave you is the smallest query, it is using two views and both
the views I have changed by using subqueries instead of CTEs. When I join
these two views, it is not getting completed at all.
Explain analyze plan for view s_v_f_promotion_history_email:
http://explain.depesz.com/s/ure Explain analyze plan for view
s_v_d_campaign_hierarchy : http://explain.depesz.com/s/WxI
Regards,
Vivek
-----Original Message-----
From: Tomas Vondra [mailto:tomas.vondra@2ndquadrant.com]
Sent: Tuesday, March 17, 2015 5:15 PM
To: vjoshi@zetainteractive.com; Jim Nasby; Scott Marlowe; Varadharajan
Mukundan
Cc: pgsql-performance@postgresql.org
Subject: Re: [PERFORM] Performance issues
On 17.3.2015 12:07, Vivekanand Joshi wrote:
> EXPLAIN ANALYZE didn't give result even after three hours.
In that case the only thing you can do is 'slice' the query into smaller
parts (representing subtrees of the plan), and analyze those first. Look for
misestimates (significant differences between estimated and actual row
counts, and very expensive parts).
We can't do that, because we don't have your data or queries, and without
the explain analyze it's difficult to give advices.
--
Tomas Vondra http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
Attachments:
Explain.sql (text/plain, 11.3K) download
s_v_f_promotion_history_email
Hash Left Join (cost=34679.90..37396.37 rows=11644 width=148) (actual time=609.472..9070.675 rows=4559289 loops=1)
Hash Cond: ((base.promo_hist_id = email.promo_hist_id) AND (base.audience_member_id = email.audience_member_id))
-> Nested Loop (cost=32782.62..35004.21 rows=11644 width=74) (actual time=567.441..4096.969 rows=4559289 loops=1)
Join Filter: (s_f_touchpoint_execution_status_history.touchpoint_execution_id = base.touchpoint_execution_id)
-> Nested Loop (cost=32782.19..34504.16 rows=1 width=16) (actual time=337.484..884.438 rows=46454 loops=1)
-> Nested Loop (cost=32781.90..34503.83 rows=1 width=24) (actual time=337.462..682.943 rows=71892 loops=1)
-> Unique (cost=32781.61..34495.50 rows=1 width=8) (actual time=337.428..478.619 rows=76088 loops=1)
-> Merge Join (cost=32781.61..34495.50 rows=1 width=8) (actual time=337.427..454.249 rows=77090 loops=1)
Merge Cond: ((s_f_touchpoint_execution_status_history.touchpoint_execution_id = s_f_touchpoint_execution_status_history_1.touchpoint_ex
ecution_id) AND (s_f_touchpoint_execution_status_history.creation_dt = (max(s_f_touchpoint_execution_status_history_1.creation_dt))))
-> Sort (cost=19697.87..20098.14 rows=160107 width=16) (actual time=132.938..155.325 rows=160898 loops=1)
Sort Key: s_f_touchpoint_execution_status_history.touchpoint_execution_id, s_f_touchpoint_execution_status_history.creation_dt
Sort Method: quicksort Memory: 13687kB
-> Seq Scan on s_f_touchpoint_execution_status_history (cost=0.00..5857.68 rows=160107 width=16) (actual time=0.012..60.421 row
s=160898 loops=1)
Filter: (touchpoint_execution_status_type_id = ANY ('{3,4,6}'::integer[]))
Rows Removed by Filter: 130206
-> Sort (cost=13083.74..13254.76 rows=68410 width=16) (actual time=204.476..227.483 rows=77456 loops=1)
Sort Key: s_f_touchpoint_execution_status_history_1.touchpoint_execution_id, (max(s_f_touchpoint_execution_status_history_1.creat
ion_dt))
Sort Method: quicksort Memory: 6656kB
-> HashAggregate (cost=6221.56..6905.66 rows=68410 width=16) (actual time=140.992..169.426 rows=76454 loops=1)
-> Seq Scan on s_f_touchpoint_execution_status_history s_f_touchpoint_execution_status_history_1 (cost=0.00..4766.04 rows
=291104 width=16) (actual time=0.013..39.851 rows=291104 loops=1)
-> Index Scan using idx_s_d_touchpoint_execution_id on s_d_touchpoint_execution tpe (cost=0.29..8.31 rows=1 width=16) (actual time=0.002..0.002 r
ows=1 loops=76088)
Index Cond: (touchpoint_execution_id = s_f_touchpoint_execution_status_history.touchpoint_execution_id)
-> Index Only Scan using s_d_touchpoint_pkey on s_d_touchpoint tp (cost=0.29..0.32 rows=1 width=8) (actual time=0.002..0.002 rows=1 loops=71892)
Index Cond: ((touchpoint_id = tpe.touchpoint_id) AND (channel_type_id = 1))
Heap Fetches: 46454
-> Index Scan using idx_pr_history on s_f_promotion_history base (cost=0.43..311.52 rows=15083 width=74) (actual time=0.002..0.033 rows=98 loops=46454)
Index Cond: (touchpoint_execution_id = tpe.touchpoint_execution_id)
-> Hash (cost=1192.71..1192.71 rows=46971 width=90) (actual time=41.978..41.978 rows=46971 loops=1)
Buckets: 8192 Batches: 1 Memory Usage: 4775kB
-> Seq Scan on s_f_promotion_history_email email (cost=0.00..1192.71 rows=46971 width=90) (actual time=0.010..19.620 rows=46971 loops=1)
s_v_d_campaign_hierarchy
Nested Loop Left Join (cost=18485.07..20366.44 rows=1 width=894) (actual time=2656.451..4041.952 rows=45360 loops=1)
-> Nested Loop (cost=18484.94..20366.29 rows=1 width=776) (actual time=2445.487..3741.049 rows=45360 loops=1)
-> Nested Loop Left Join (cost=18484.81..20366.12 rows=1 width=658) (actual time=2445.471..3648.007 rows=48926 loops=1)
-> Hash Join (cost=18484.68..20365.96 rows=1 width=340) (actual time=2267.997..3359.148 rows=48926 loops=1)
Hash Cond: ((s_f_touchpoint_execution_status_history_1.touchpoint_execution_id = s_f_touchpoint_execution_status_history.touchpoint_execution_id) A
ND ((max(s_f_touchpoint_execution_status_history_1.creation_dt)) = s_f_touchpoint_execution_status_history.creation_dt))
-> HashAggregate (cost=6221.56..6905.66 rows=68410 width=16) (actual time=140.079..181.465 rows=76454 loops=1)
-> Seq Scan on s_f_touchpoint_execution_status_history s_f_touchpoint_execution_status_history_1 (cost=0.00..4766.04 rows=291104 width=16)
(actual time=0.010..38.894 rows=291104 loops=1)
-> Hash (cost=12263.09..12263.09 rows=2 width=356) (actual time=2127.869..2127.869 rows=131427 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 37990kB
-> Hash Join (cost=6259.72..12263.09 rows=2 width=356) (actual time=1357.652..1990.177 rows=131427 loops=1)
Hash Cond: (s_f_touchpoint_execution_status_history.touchpoint_execution_id = tp_exec.touchpoint_execution_id)
-> Seq Scan on s_f_touchpoint_execution_status_history (cost=0.00..5493.80 rows=135878 width=16) (actual time=0.011..52.932 rows=1362
80 loops=1)
Filter: (touchpoint_execution_status_type_id = ANY ('{3,4}'::integer[]))
Rows Removed by Filter: 154824
-> Hash (cost=6259.71..6259.71 rows=1 width=340) (actual time=1357.605..1357.605 rows=72427 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 19510kB
-> Nested Loop Left Join (cost=1955.40..6259.71 rows=1 width=340) (actual time=158.779..1269.938 rows=72427 loops=1)
-> Nested Loop Left Join (cost=1955.27..6259.55 rows=1 width=222) (actual time=31.836..998.297 rows=72427 loops=1)
-> Nested Loop (cost=1954.99..6259.24 rows=1 width=197) (actual time=31.826..829.278 rows=72427 loops=1)
-> Nested Loop (cost=1954.71..6258.92 rows=1 width=173) (actual time=31.816..636.122 rows=72427 loops=1)
Join Filter: (camp_exec.campaign_id = wave.campaign_id)
Rows Removed by Join Filter: 243
-> Nested Loop (cost=1954.42..6254.67 rows=13 width=167) (actual time=31.804..435.461 rows=72670 loops=
1)
-> Hash Join (cost=1954.13..6249.67 rows=13 width=108) (actual time=31.777..210.346 rows=72670 lo
ops=1)
Hash Cond: ((tp_exec.touchpoint_id = tp.touchpoint_id) AND (wave_exec.wave_id = tp.wave_id))
-> Hash Join (cost=1576.83..4595.51 rows=72956 width=90) (actual time=25.984..141.564 rows=
72956 loops=1)
Hash Cond: (tp_exec.wave_execution_id = wave_exec.wave_execution_id)
-> Seq Scan on s_d_touchpoint_execution tp_exec (cost=0.00..1559.56 rows=72956 width=
42) (actual time=0.005..14.945 rows=72956 loops=1)
-> Hash (cost=1001.37..1001.37 rows=46037 width=56) (actual time=25.922..25.922 rows=
46037 loops=1)
Buckets: 8192 Batches: 1 Memory Usage: 4104kB
-> Seq Scan on s_d_wave_execution wave_exec (cost=0.00..1001.37 rows=46037 widt
h=56) (actual time=0.007..10.424 rows=46037 loops=1)
-> Hash (cost=212.72..212.72 rows=10972 width=26) (actual time=5.773..5.773 rows=10972 loop
s=1)
Buckets: 2048 Batches: 1 Memory Usage: 645kB
-> Seq Scan on s_d_touchpoint tp (cost=0.00..212.72 rows=10972 width=26) (actual time
=0.007..2.350 rows=10972 loops=1)
-> Index Scan using s_d_campaign_execution_idx on s_d_campaign_execution camp_exec (cost=0.29..0.
37 rows=1 width=67) (actual time=0.002..0.002 rows=1 loops=72670)
Index Cond: (campaign_execution_id = wave_exec.campaign_execution_id)
-> Index Scan using s_d_wave_pkey on s_d_wave wave (cost=0.29..0.31 rows=1 width=22) (actual time=0.002
..0.002 rows=1 loops=72670)
Index Cond: (wave_id = wave_exec.wave_id)
-> Index Scan using s_d_campaign_pkey on s_d_campaign camp (cost=0.29..0.32 rows=1 width=40) (actual time=0.0
02..0.002 rows=1 loops=72427)
Index Cond: (campaign_id = camp_exec.campaign_id)
-> Index Scan using s_d_content_pkey on s_d_content content (cost=0.28..0.30 rows=1 width=33) (actual time=0.001..0
.002 rows=1 loops=72427)
Index Cond: (tp_exec.content_id = content_id)
-> Index Scan using s_d_message_type_pkey on s_d_message_type message_type (cost=0.13..0.15 rows=1 width=120) (actual tim
e=0.003..0.003 rows=1 loops=72427)
Index Cond: (tp_exec.message_type_id = message_type_id)
-> Index Scan using s_d_group_pkey on s_d_group grup (cost=0.13..0.15 rows=1 width=320) (actual time=0.005..0.005 rows=1 loops=48926)
Index Cond: (camp_exec.group_id = group_id)
-> Index Scan using d_channel_pk on s_d_channel_type channel (cost=0.13..0.15 rows=1 width=120) (actual time=0.001..0.001 rows=1 loops=48926)
Index Cond: (channel_type_id = tp.channel_type_id)
-> Index Scan using s_d_category_pkey on s_d_category "CATEGORY" (cost=0.13..0.15 rows=1 width=120) (actual time=0.006..0.006 rows=1 loops=45360)
Index Cond: (camp.category_id = category_id)
* Re: Performance issues
2015-03-13 11:44 Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 12:58 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 13:03 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 13:10 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 19:59 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 20:36 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-13 20:46 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 22:03 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 22:25 ` Re: Performance issues Scott Marlowe <scott.marlowe@gmail.com>
2015-03-13 23:28 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-14 00:12 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-16 19:43 ` Re: Performance issues Jim Nasby <Jim.Nasby@BlueTreble.com>
2015-03-16 20:59 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 00:06 ` Re: Performance issues Jim Nasby <Jim.Nasby@BlueTreble.com>
2015-03-17 11:07 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-17 11:45 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 12:10 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
@ 2015-03-17 13:55 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 14:19 ` Re: Performance issues Thomas Kellerer <spam_eater@gmx.net>
0 siblings, 1 reply; 59+ messages in thread
From: Tomas Vondra @ 2015-03-17 13:55 UTC (permalink / raw)
To: vjoshi@zetainteractive.com, Jim Nasby <Jim.Nasby@bluetreble.com>; Scott Marlowe <scott.marlowe@gmail.com>; Varadharajan Mukundan <srinathsmn@gmail.com>; +Cc: pgsql-performance
Just as I feared, the attached explain analyze results show significant
misestimates, like this for example:
Nested Loop (cost=32782.19..34504.16 rows=1 width=16)
(actual time=337.484..884.438 rows=46454 loops=1)
Nested Loop (cost=18484.94..20366.29 rows=1 width=776)
(actual time=2445.487..3741.049 rows=45360 loops=1)
Hash Left Join (cost=34679.90..37396.37 rows=11644 width=148)
(actual time=609.472..9070.675 rows=4559289 loops=1)
There's plenty of nested loop joins - the optimizer believes there will
be only a few rows in the outer relation, but gets order of magnitude
more tuples. And nested loops are terrible in that case.
In case of the first view, it seems to be caused by this:
Merge Cond:
((s_f_touchpoint_execution_status_history.touchpoint_execution_id =
s_f_touchpoint_execution_status_history_1.touchpoint_ex
ecution_id) AND (s_f_touchpoint_execution_status_history.creation_dt =
(max(s_f_touchpoint_execution_status_history_1.creation_dt))))
especially the ':id = max(:id)' condition is probably giving the
optimizer a hard time. This is a conceptually difficult poblem (i.e.
fixing this at the optimizer level is unlikely to happen any time soon,
because it effectively means you have to predict the statistical
properties of the aggregation).
You may try increasing the statistical target, which makes the stats a
bit more detailed (the default on 9.4 is 100):
SET default_statistics_target = 10000;
ANALYZE;
But I don't really believe this might really fix the problem.
But maybe it's possible to rewrite the query somehow?
Let's experiment a bit - remove the aggregation, i.e. join directly to
s_f_touchpoint_execution_status_history. It'll return wrong results, but
the estimates should be better, so let's see what happens.
You may also try disabling nested loops - the other join algorithms
usually perform better with large row counts.
SET enable_nestloop = false;
This is not a production-suitable solution, but for experimenting that's OK.
ISTM what the aggregation (or the whole mergejoin) does is selecting the
last s_f_touchpoint_execution_status_history record for each
touchpoint_execution_id.
There are better ways to determine that, IMHO. For example:
(1) adding a 'is_last' flag to s_f_touchpoint_execution_status_history
This however requires maintaining that flag somehow, but the join
would not be needed at all.
The "last IDs" might be maintained in a separate table - the join
would be still necessary, but it might be less intrusive and
cheper to maintain.
(2) using window functions, e.g. like this:
SELECT * FROM (
SELECT *,
ROW_NUMBER() OVER (PARTITION BY touchpoint_execution_id
ORDER BY FROM max_creation_dt) AS rn
FROM s_f_touchpoint_execution_status_history
) foo WHERE rn = 1
But estimating this is also rather difficult ...
(3) Using temporary table / MV - this really depends on your
requirements, load schedule, how you run the queries etc. It would
however fix the estimation errors (probably).
The 2nd view seems to suffer because of the same issue (underestimates
leading to choice of nested loops), but caused by something else:
-> Hash Join (cost=1954.13..6249.67 rows=13 width=108)
(actual time=31.777..210.346 rows=72670 loops=1)
Hash Cond: ((tp_exec.touchpoint_id = tp.touchpoint_id)
AND (wave_exec.wave_id = tp.wave_id))
Estimating cardinality of joins with multi-column conditions is
difficult, no idea how to fix that at the moment.
--
Tomas Vondra http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Performance issues
2015-03-13 11:44 Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 12:58 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 13:03 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 13:10 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 19:59 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 20:36 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-13 20:46 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 22:03 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 22:25 ` Re: Performance issues Scott Marlowe <scott.marlowe@gmail.com>
2015-03-13 23:28 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-14 00:12 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-16 19:43 ` Re: Performance issues Jim Nasby <Jim.Nasby@BlueTreble.com>
2015-03-16 20:59 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 00:06 ` Re: Performance issues Jim Nasby <Jim.Nasby@BlueTreble.com>
2015-03-17 11:07 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-17 11:45 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 12:10 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-17 13:55 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
@ 2015-03-17 14:19 ` Thomas Kellerer <spam_eater@gmx.net>
2015-03-17 14:43 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 1 reply; 59+ messages in thread
From: Thomas Kellerer @ 2015-03-17 14:19 UTC (permalink / raw)
To: pgsql-performance
Tomas Vondra schrieb am 17.03.2015 um 14:55:
> (2) using window functions, e.g. like this:
>
> SELECT * FROM (
> SELECT *,
> ROW_NUMBER() OVER (PARTITION BY touchpoint_execution_id
> ORDER BY FROM max_creation_dt) AS rn
> FROM s_f_touchpoint_execution_status_history
> ) foo WHERE rn = 1
>
> But estimating this is also rather difficult ...
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Performance issues
2015-03-13 11:44 Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 12:58 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 13:03 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 13:10 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 19:59 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 20:36 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-13 20:46 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 22:03 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 22:25 ` Re: Performance issues Scott Marlowe <scott.marlowe@gmail.com>
2015-03-13 23:28 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-14 00:12 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-16 19:43 ` Re: Performance issues Jim Nasby <Jim.Nasby@BlueTreble.com>
2015-03-16 20:59 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 00:06 ` Re: Performance issues Jim Nasby <Jim.Nasby@BlueTreble.com>
2015-03-17 11:07 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-17 11:45 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 12:10 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-17 13:55 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 14:19 ` Re: Performance issues Thomas Kellerer <spam_eater@gmx.net>
@ 2015-03-17 14:43 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 15:10 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-17 15:16 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-17 15:24 ` Re: Performance issues Thomas Kellerer <spam_eater@gmx.net>
0 siblings, 3 replies; 59+ messages in thread
From: Tomas Vondra @ 2015-03-17 14:43 UTC (permalink / raw)
To: pgsql-performance
On 17.3.2015 15:19, Thomas Kellerer wrote:
> Tomas Vondra schrieb am 17.03.2015 um 14:55:
>> (2) using window functions, e.g. like this:
>>
>> SELECT * FROM (
>> SELECT *,
>> ROW_NUMBER() OVER (PARTITION BY touchpoint_execution_id
>> ORDER BY FROM max_creation_dt) AS rn
>> FROM s_f_touchpoint_execution_status_history
>> ) foo WHERE rn = 1
>>
>> But estimating this is also rather difficult ...
>
>
> From my experience rewriting something like the above using DISTINCT
> ON is usually faster.
How do you get the last record (with respect to a timestamp column)
using a DISTINCT ON?
--
Tomas Vondra http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Performance issues
2015-03-13 11:44 Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 12:58 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 13:03 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 13:10 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 19:59 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 20:36 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-13 20:46 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 22:03 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 22:25 ` Re: Performance issues Scott Marlowe <scott.marlowe@gmail.com>
2015-03-13 23:28 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-14 00:12 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-16 19:43 ` Re: Performance issues Jim Nasby <Jim.Nasby@BlueTreble.com>
2015-03-16 20:59 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 00:06 ` Re: Performance issues Jim Nasby <Jim.Nasby@BlueTreble.com>
2015-03-17 11:07 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-17 11:45 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 12:10 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-17 13:55 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 14:19 ` Re: Performance issues Thomas Kellerer <spam_eater@gmx.net>
2015-03-17 14:43 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
@ 2015-03-17 15:10 ` Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-17 15:28 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2 siblings, 1 reply; 59+ messages in thread
From: Vivekanand Joshi @ 2015-03-17 15:10 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; pgsql-performance
The confusion for me here is that :
I am getting results from the view in around 3 seconds
(S_V_D_CAMPAIGN_HIERARCHY) and 25 seconds (S_V_F_PROMOTION_HISTORY_EMAIL)
But when I am using these two views in the query as the joining tables, it
doesn't give any result. As per my understanding, the planner is making new
plan and that is costly instead of using output from the view, which is
actually understandable.
Is there a way, we can do anything about it?
I hope I am making some sense here.
Regards,
Vivek
-----Original Message-----
From: pgsql-performance-owner@postgresql.org
[mailto:pgsql-performance-owner@postgresql.org] On Behalf Of Tomas Vondra
Sent: Tuesday, March 17, 2015 8:13 PM
To: pgsql-performance@postgresql.org
Subject: Re: [PERFORM] Performance issues
On 17.3.2015 15:19, Thomas Kellerer wrote:
> Tomas Vondra schrieb am 17.03.2015 um 14:55:
>> (2) using window functions, e.g. like this:
>>
>> SELECT * FROM (
>> SELECT *,
>> ROW_NUMBER() OVER (PARTITION BY touchpoint_execution_id
>> ORDER BY FROM max_creation_dt) AS rn
>> FROM s_f_touchpoint_execution_status_history
>> ) foo WHERE rn = 1
>>
>> But estimating this is also rather difficult ...
>
>
> From my experience rewriting something like the above using DISTINCT
> ON is usually faster.
How do you get the last record (with respect to a timestamp column) using a
DISTINCT ON?
--
Tomas Vondra http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Performance issues
2015-03-13 11:44 Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 12:58 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 13:03 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 13:10 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 19:59 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 20:36 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-13 20:46 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 22:03 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 22:25 ` Re: Performance issues Scott Marlowe <scott.marlowe@gmail.com>
2015-03-13 23:28 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-14 00:12 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-16 19:43 ` Re: Performance issues Jim Nasby <Jim.Nasby@BlueTreble.com>
2015-03-16 20:59 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 00:06 ` Re: Performance issues Jim Nasby <Jim.Nasby@BlueTreble.com>
2015-03-17 11:07 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-17 11:45 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 12:10 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-17 13:55 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 14:19 ` Re: Performance issues Thomas Kellerer <spam_eater@gmx.net>
2015-03-17 14:43 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 15:10 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
@ 2015-03-17 15:28 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 0 replies; 59+ messages in thread
From: Tomas Vondra @ 2015-03-17 15:28 UTC (permalink / raw)
To: vjoshi@zetainteractive.com; pgsql-performance
On 17.3.2015 16:10, Vivekanand Joshi wrote:
> The confusion for me here is that :
>
>
> I am getting results from the view in around 3 seconds
> (S_V_D_CAMPAIGN_HIERARCHY) and 25 seconds (S_V_F_PROMOTION_HISTORY_EMAIL)
>
> But when I am using these two views in the query as the joining
> tables, it doesn't give any result. As per my understanding, the
> planner is making new plan and that is costly instead of using
> output from the view, which is actually understandable.
In general, yes. The problem is that the plan is constructed based on
the estimates, and those are very inaccurate in this case.
The planner may do various changes, but let's assume that does not
happen and the plans are executed and and the results are joined.
For example what might happen is this:
for each row in 's_v_d_campaign_hierarchy' (1 row expected):
execute s_v_f_promotion_history_email & join (11644 rows exp.)
But then it gets 45k rows from s_v_d_campaign_hierarchy, and ~400x more
rows from s_v_f_promotion_history_email (I'm neglecting the join
condition here, but that's not really significant). Kaboooom!
In reality, the plan is reorganized (e.g. different join order), but the
misestimates are still lurking there.
> Is there a way, we can do anything about it?
Rephrasing the query so that the planner can estimate it more accurately.
--
Tomas Vondra http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Performance issues
2015-03-13 11:44 Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 12:58 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 13:03 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 13:10 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 19:59 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 20:36 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-13 20:46 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 22:03 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 22:25 ` Re: Performance issues Scott Marlowe <scott.marlowe@gmail.com>
2015-03-13 23:28 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-14 00:12 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-16 19:43 ` Re: Performance issues Jim Nasby <Jim.Nasby@BlueTreble.com>
2015-03-16 20:59 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 00:06 ` Re: Performance issues Jim Nasby <Jim.Nasby@BlueTreble.com>
2015-03-17 11:07 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-17 11:45 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 12:10 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-17 13:55 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 14:19 ` Re: Performance issues Thomas Kellerer <spam_eater@gmx.net>
2015-03-17 14:43 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
@ 2015-03-17 15:16 ` Vivekanand Joshi <vjoshi@zetainteractive.com>
2 siblings, 0 replies; 59+ messages in thread
From: Vivekanand Joshi @ 2015-03-17 15:16 UTC (permalink / raw)
To: vjoshi@zetainteractive.com, Tomas Vondra <tomas.vondra@2ndquadrant.com>; pgsql-performance
This is the explain for a simple query:
explain Select * from S_V_F_PROMOTION_HISTORY_EMAIL a inner join
S_V_D_CAMPAIGN_HIERARCHY b on a.touchpoint_execution_id =
b.touchpoint_execution_id;
http://explain.depesz.com/s/gse
I am wondering the total cost here is less even then the result is not
coming out.
Regards,
Vivek
-----Original Message-----
From: Vivekanand Joshi [mailto:vjoshi@zetainteractive.com]
Sent: Tuesday, March 17, 2015 8:40 PM
To: 'Tomas Vondra'; 'pgsql-performance@postgresql.org'
Subject: RE: [PERFORM] Performance issues
The confusion for me here is that :
I am getting results from the view in around 3 seconds
(S_V_D_CAMPAIGN_HIERARCHY) and 25 seconds (S_V_F_PROMOTION_HISTORY_EMAIL)
But when I am using these two views in the query as the joining tables, it
doesn't give any result. As per my understanding, the planner is making new
plan and that is costly instead of using output from the view, which is
actually understandable.
Is there a way, we can do anything about it?
I hope I am making some sense here.
Regards,
Vivek
-----Original Message-----
From: pgsql-performance-owner@postgresql.org
[mailto:pgsql-performance-owner@postgresql.org] On Behalf Of Tomas Vondra
Sent: Tuesday, March 17, 2015 8:13 PM
To: pgsql-performance@postgresql.org
Subject: Re: [PERFORM] Performance issues
On 17.3.2015 15:19, Thomas Kellerer wrote:
> Tomas Vondra schrieb am 17.03.2015 um 14:55:
>> (2) using window functions, e.g. like this:
>>
>> SELECT * FROM (
>> SELECT *,
>> ROW_NUMBER() OVER (PARTITION BY touchpoint_execution_id
>> ORDER BY FROM max_creation_dt) AS rn
>> FROM s_f_touchpoint_execution_status_history
>> ) foo WHERE rn = 1
>>
>> But estimating this is also rather difficult ...
>
>
> From my experience rewriting something like the above using DISTINCT
> ON is usually faster.
How do you get the last record (with respect to a timestamp column) using a
DISTINCT ON?
--
Tomas Vondra http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
Attachments:
Explain.sql (text/plain, 9.4K) download
Nested Loop (cost=41187.21..45053.54 rows=1 width=403)
Join Filter: (s_f_touchpoint_execution_status_history_2.touchpoint_execution_id = s_f_touchpoint_execution_status_history.touchpoint_execution_id)
-> Nested Loop Left Join (cost=18488.74..20591.32 rows=1 width=271)
-> Nested Loop (cost=18488.61..20591.17 rows=1 width=263)
-> Nested Loop Left Join (cost=18488.48..20591.00 rows=1 width=255)
-> Hash Join (cost=18488.35..20590.84 rows=1 width=247)
Hash Cond: ((s_f_touchpoint_execution_status_history_1.touchpoint_execution_id = s_f_touchpoint_execution_status_history.touchpoint_execution
_id) AND ((max(s_f_touchpoint_execution_status_history_1.creation_dt)) = s_f_touchpoint_execution_status_history.creation_dt))
-> HashAggregate (cost=6221.56..6986.10 rows=76454 width=16)
-> Seq Scan on s_f_touchpoint_execution_status_history s_f_touchpoint_execution_status_history_1 (cost=0.00..4766.04 rows=291104 widt
h=16)
-> Hash (cost=12266.76..12266.76 rows=2 width=247)
-> Hash Join (cost=6261.89..12266.76 rows=2 width=247)
Hash Cond: (s_f_touchpoint_execution_status_history.touchpoint_execution_id = tp_exec.touchpoint_execution_id)
-> Seq Scan on s_f_touchpoint_execution_status_history (cost=0.00..5493.80 rows=136280 width=16)
Filter: (touchpoint_execution_status_type_id = ANY ('{3,4}'::integer[]))
-> Hash (cost=6261.88..6261.88 rows=1 width=231)
-> Nested Loop Left Join (cost=1955.40..6261.88 rows=1 width=231)
-> Nested Loop Left Join (cost=1955.27..6261.72 rows=1 width=222)
-> Nested Loop (cost=1954.99..6261.41 rows=1 width=197)
-> Nested Loop (cost=1954.71..6261.08 rows=1 width=173)
Join Filter: (camp_exec.campaign_id = wave.campaign_id)
-> Nested Loop (cost=1954.42..6255.85 rows=16 width=167)
-> Hash Join (cost=1954.13..6249.70 rows=16 width=108)
Hash Cond: ((tp_exec.touchpoint_id = tp_1.touchpoint_id) AND (wave_exec.wave_id = tp_1.
wave_id))
-> Hash Join (cost=1576.83..4595.51 rows=72956 width=90)
Hash Cond: (tp_exec.wave_execution_id = wave_exec.wave_execution_id)
-> Seq Scan on s_d_touchpoint_execution tp_exec (cost=0.00..1559.56 rows=72956
width=42)
-> Hash (cost=1001.37..1001.37 rows=46037 width=56)
-> Seq Scan on s_d_wave_execution wave_exec (cost=0.00..1001.37 rows=4603
7 width=56)
-> Hash (cost=212.72..212.72 rows=10972 width=26)
-> Seq Scan on s_d_touchpoint tp_1 (cost=0.00..212.72 rows=10972 width=26)
-> Index Scan using s_d_campaign_execution_idx on s_d_campaign_execution camp_exec (cost=0.
29..0.37 rows=1 width=67)
Index Cond: (campaign_execution_id = wave_exec.campaign_execution_id)
-> Index Scan using s_d_wave_pkey on s_d_wave wave (cost=0.29..0.31 rows=1 width=22)
Index Cond: (wave_id = wave_exec.wave_id)
-> Index Scan using s_d_campaign_pkey on s_d_campaign camp (cost=0.29..0.32 rows=1 width=40)
Index Cond: (campaign_id = camp_exec.campaign_id)
-> Index Scan using s_d_content_pkey on s_d_content content (cost=0.28..0.30 rows=1 width=33)
Index Cond: (tp_exec.content_id = content_id)
-> Index Scan using s_d_message_type_pkey on s_d_message_type message_type (cost=0.13..0.15 rows=1 width=11)
Index Cond: (tp_exec.message_type_id = message_type_id)
-> Index Scan using s_d_group_pkey on s_d_group grup (cost=0.13..0.15 rows=1 width=10)
Index Cond: (camp_exec.group_id = group_id)
-> Index Scan using d_channel_pk on s_d_channel_type channel (cost=0.13..0.15 rows=1 width=10)
Index Cond: (channel_type_id = tp_1.channel_type_id)
-> Index Scan using s_d_category_pkey on s_d_category "CATEGORY" (cost=0.13..0.15 rows=1 width=10)
Index Cond: (camp.category_id = category_id)
-> Hash Left Join (cost=22698.47..23979.56 rows=38611 width=164)
Hash Cond: ((base.promo_hist_id = email.promo_hist_id) AND (base.audience_member_id = email.audience_member_id))
-> Nested Loop (cost=20801.19..21310.02 rows=38611 width=90)
Join Filter: (s_f_touchpoint_execution_status_history_2.touchpoint_execution_id = base.touchpoint_execution_id)
-> Nested Loop (cost=20800.76..20808.85 rows=1 width=16)
-> Nested Loop (cost=20800.47..20808.52 rows=1 width=24)
-> HashAggregate (cost=20800.18..20800.19 rows=1 width=8)
-> Hash Join (cost=14492.71..20800.18 rows=1 width=8)
Hash Cond: ((s_f_touchpoint_execution_status_history_1_1.touchpoint_execution_id = s_f_touchpoint_execution_status_history_2.touc
hpoint_execution_id) AND ((max(s_f_touchpoint_execution_status_history_1_1.creation_dt)) = s_f_touchpoint_execution_status_history_2.creation_dt))
-> HashAggregate (cost=6221.56..6986.10 rows=76454 width=16)
-> Seq Scan on s_f_touchpoint_execution_status_history s_f_touchpoint_execution_status_history_1_1 (cost=0.00..4766.04 ro
ws=291104 width=16)
-> Hash (cost=5857.68..5857.68 rows=160898 width=16)
-> Seq Scan on s_f_touchpoint_execution_status_history s_f_touchpoint_execution_status_history_2 (cost=0.00..5857.68 rows
=160898 width=16)
Filter: (touchpoint_execution_status_type_id = ANY ('{3,4,6}'::integer[]))
-> Index Scan using idx_s_d_touchpoint_execution_id on s_d_touchpoint_execution tpe (cost=0.29..8.31 rows=1 width=16)
Index Cond: (touchpoint_execution_id = s_f_touchpoint_execution_status_history_2.touchpoint_execution_id)
-> Index Only Scan using s_d_touchpoint_pkey on s_d_touchpoint tp (cost=0.29..0.32 rows=1 width=8)
Index Cond: ((touchpoint_id = tpe.touchpoint_id) AND (channel_type_id = 1))
-> Append (cost=0.43..312.44 rows=15099 width=74)
-> Index Scan using idx_pr_history on s_f_promotion_history base (cost=0.43..311.52 rows=15083 width=74)
Index Cond: (touchpoint_execution_id = tpe.touchpoint_execution_id)
-> Index Scan using s_f_promotion_history_table1_teiidx on s_f_promotion_history_table1 base_1 (cost=0.15..0.23 rows=4 width=74)
Index Cond: (touchpoint_execution_id = tpe.touchpoint_execution_id)
-> Index Scan using s_f_promotion_history_table2_teiidx on s_f_promotion_history_table2 base_2 (cost=0.15..0.23 rows=4 width=74)
Index Cond: (touchpoint_execution_id = tpe.touchpoint_execution_id)
-> Index Scan using s_f_promotion_history_table3_teiidx on s_f_promotion_history_table3 base_3 (cost=0.15..0.23 rows=4 width=74)
Index Cond: (touchpoint_execution_id = tpe.touchpoint_execution_id)
-> Index Scan using s_f_promotion_history_table4_teiidx on s_f_promotion_history_table4 base_4 (cost=0.15..0.23 rows=4 width=74)
Index Cond: (touchpoint_execution_id = tpe.touchpoint_execution_id)
-> Hash (cost=1192.71..1192.71 rows=46971 width=90)
-> Seq Scan on s_f_promotion_history_email email (cost=0.00..1192.71 rows=46971 width=90)
* Re: Performance issues
2015-03-13 11:44 Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 12:58 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 13:03 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 13:10 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 19:59 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 20:36 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-13 20:46 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 22:03 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 22:25 ` Re: Performance issues Scott Marlowe <scott.marlowe@gmail.com>
2015-03-13 23:28 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-14 00:12 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-16 19:43 ` Re: Performance issues Jim Nasby <Jim.Nasby@BlueTreble.com>
2015-03-16 20:59 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 00:06 ` Re: Performance issues Jim Nasby <Jim.Nasby@BlueTreble.com>
2015-03-17 11:07 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-17 11:45 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 12:10 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-17 13:55 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 14:19 ` Re: Performance issues Thomas Kellerer <spam_eater@gmx.net>
2015-03-17 14:43 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
@ 2015-03-17 15:24 ` Thomas Kellerer <spam_eater@gmx.net>
2015-03-17 15:30 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-18 17:31 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2 siblings, 2 replies; 59+ messages in thread
From: Thomas Kellerer @ 2015-03-17 15:24 UTC (permalink / raw)
To: pgsql-performance
Tomas Vondra schrieb am 17.03.2015 um 15:43:
> On 17.3.2015 15:19, Thomas Kellerer wrote:
>> Tomas Vondra schrieb am 17.03.2015 um 14:55:
>>> (2) using window functions, e.g. like this:
>>>
>>> SELECT * FROM (
>>> SELECT *,
>>> ROW_NUMBER() OVER (PARTITION BY touchpoint_execution_id
>>> ORDER BY FROM max_creation_dt) AS rn
>>> FROM s_f_touchpoint_execution_status_history
>>> ) foo WHERE rn = 1
>>>
>>> But estimating this is also rather difficult ...
>>
>>
>> From my experience rewriting something like the above using DISTINCT
>> ON is usually faster.
>
> How do you get the last record (with respect to a timestamp column)
> using a DISTINCT ON?
You need to use "order by ... desc". See here: http://sqlfiddle.com/#!15/d4846/2
Btw: your row_number() usage wouldn't return the "latest" row either.
It would return the "oldest" row.
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Performance issues
2015-03-13 11:44 Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 12:58 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 13:03 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 13:10 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 19:59 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 20:36 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-13 20:46 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 22:03 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 22:25 ` Re: Performance issues Scott Marlowe <scott.marlowe@gmail.com>
2015-03-13 23:28 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-14 00:12 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-16 19:43 ` Re: Performance issues Jim Nasby <Jim.Nasby@BlueTreble.com>
2015-03-16 20:59 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 00:06 ` Re: Performance issues Jim Nasby <Jim.Nasby@BlueTreble.com>
2015-03-17 11:07 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-17 11:45 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 12:10 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-17 13:55 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 14:19 ` Re: Performance issues Thomas Kellerer <spam_eater@gmx.net>
2015-03-17 14:43 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 15:24 ` Re: Performance issues Thomas Kellerer <spam_eater@gmx.net>
@ 2015-03-17 15:30 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
1 sibling, 0 replies; 59+ messages in thread
From: Tomas Vondra @ 2015-03-17 15:30 UTC (permalink / raw)
To: pgsql-performance
On 17.3.2015 16:24, Thomas Kellerer wrote:
> Tomas Vondra schrieb am 17.03.2015 um 15:43:
>> On 17.3.2015 15:19, Thomas Kellerer wrote:
>>> Tomas Vondra schrieb am 17.03.2015 um 14:55:
>>>> (2) using window functions, e.g. like this:
>>>>
>>>> SELECT * FROM (
>>>> SELECT *,
>>>> ROW_NUMBER() OVER (PARTITION BY touchpoint_execution_id
>>>> ORDER BY FROM max_creation_dt) AS rn
>>>> FROM s_f_touchpoint_execution_status_history
>>>> ) foo WHERE rn = 1
>>>>
>>>> But estimating this is also rather difficult ...
>>>
>>>
>>> From my experience rewriting something like the above using DISTINCT
>>> ON is usually faster.
>>
>> How do you get the last record (with respect to a timestamp column)
>> using a DISTINCT ON?
>
> You need to use "order by ... desc". See here: http://sqlfiddle.com/#!15/d4846/2
Nice, thanks!
>
> Btw: your row_number() usage wouldn't return the "latest" row either.
> It would return the "oldest" row.
Oh, right. I forgot the DESC in the window.
--
Tomas Vondra http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Performance issues
2015-03-13 11:44 Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 12:58 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 13:03 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 13:10 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 19:59 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 20:36 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-13 20:46 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 22:03 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 22:25 ` Re: Performance issues Scott Marlowe <scott.marlowe@gmail.com>
2015-03-13 23:28 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-14 00:12 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-16 19:43 ` Re: Performance issues Jim Nasby <Jim.Nasby@BlueTreble.com>
2015-03-16 20:59 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 00:06 ` Re: Performance issues Jim Nasby <Jim.Nasby@BlueTreble.com>
2015-03-17 11:07 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-17 11:45 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 12:10 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-17 13:55 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 14:19 ` Re: Performance issues Thomas Kellerer <spam_eater@gmx.net>
2015-03-17 14:43 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 15:24 ` Re: Performance issues Thomas Kellerer <spam_eater@gmx.net>
@ 2015-03-18 17:31 ` Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-18 17:51 ` Re: Performance issues Felipe Santos <felipepts@gmail.com>
2015-03-18 18:36 ` Re: Performance issues Jerry Sievers <gsievers19@comcast.net>
2015-03-18 19:23 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
1 sibling, 3 replies; 59+ messages in thread
From: Vivekanand Joshi @ 2015-03-18 17:31 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; pgsql-performance
So, here is the first taste of success and which gives me the confidence
that if properly worked out with a good hardware and proper tuning,
PostgreSQL could be a good replacement.
Out of the 9 reports which needs to be migrated in PostgreSQL, 3 are now
running.
Report 4 was giving an issue and I will see it tomorrow.
Just to inform you guys that, the thing that helped most is setting
enable_nestloops to false worked. Plans are now not miscalculated.
But this is not a production-suitable setting. So what do you think how to
get a work around this?
Regards,
Vivek
-----Original Message-----
From: pgsql-performance-owner@postgresql.org
[mailto:pgsql-performance-owner@postgresql.org] On Behalf Of Tomas Vondra
Sent: Tuesday, March 17, 2015 9:00 PM
To: pgsql-performance@postgresql.org
Subject: Re: [PERFORM] Performance issues
On 17.3.2015 16:24, Thomas Kellerer wrote:
> Tomas Vondra schrieb am 17.03.2015 um 15:43:
>> On 17.3.2015 15:19, Thomas Kellerer wrote:
>>> Tomas Vondra schrieb am 17.03.2015 um 14:55:
>>>> (2) using window functions, e.g. like this:
>>>>
>>>> SELECT * FROM (
>>>> SELECT *,
>>>> ROW_NUMBER() OVER (PARTITION BY touchpoint_execution_id
>>>> ORDER BY FROM max_creation_dt) AS rn
>>>> FROM s_f_touchpoint_execution_status_history
>>>> ) foo WHERE rn = 1
>>>>
>>>> But estimating this is also rather difficult ...
>>>
>>>
>>> From my experience rewriting something like the above using DISTINCT
>>> ON is usually faster.
>>
>> How do you get the last record (with respect to a timestamp column)
>> using a DISTINCT ON?
>
> You need to use "order by ... desc". See here:
> http://sqlfiddle.com/#!15/d4846/2
Nice, thanks!
>
> Btw: your row_number() usage wouldn't return the "latest" row either.
> It would return the "oldest" row.
Oh, right. I forgot the DESC in the window.
--
Tomas Vondra http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Performance issues
2015-03-13 11:44 Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 12:58 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 13:03 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 13:10 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 19:59 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 20:36 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-13 20:46 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 22:03 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 22:25 ` Re: Performance issues Scott Marlowe <scott.marlowe@gmail.com>
2015-03-13 23:28 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-14 00:12 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-16 19:43 ` Re: Performance issues Jim Nasby <Jim.Nasby@BlueTreble.com>
2015-03-16 20:59 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 00:06 ` Re: Performance issues Jim Nasby <Jim.Nasby@BlueTreble.com>
2015-03-17 11:07 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-17 11:45 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 12:10 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-17 13:55 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 14:19 ` Re: Performance issues Thomas Kellerer <spam_eater@gmx.net>
2015-03-17 14:43 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 15:24 ` Re: Performance issues Thomas Kellerer <spam_eater@gmx.net>
2015-03-18 17:31 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
@ 2015-03-18 17:51 ` Felipe Santos <felipepts@gmail.com>
2 siblings, 0 replies; 59+ messages in thread
From: Felipe Santos @ 2015-03-18 17:51 UTC (permalink / raw)
To: vjoshi@zetainteractive.com; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; pgsql-performance
2015-03-18 14:31 GMT-03:00 Vivekanand Joshi <vjoshi@zetainteractive.com>:
> So, here is the first taste of success and which gives me the confidence
> that if properly worked out with a good hardware and proper tuning,
> PostgreSQL could be a good replacement.
>
> Out of the 9 reports which needs to be migrated in PostgreSQL, 3 are now
> running.
>
> Report 4 was giving an issue and I will see it tomorrow.
>
> Just to inform you guys that, the thing that helped most is setting
> enable_nestloops to false worked. Plans are now not miscalculated.
>
>
>
>
> Regards,
> Vivek
>
> -----Original Message-----
> From: pgsql-performance-owner@postgresql.org
> [mailto:pgsql-performance-owner@postgresql.org] On Behalf Of Tomas Vondra
> Sent: Tuesday, March 17, 2015 9:00 PM
> To: pgsql-performance@postgresql.org
> Subject: Re: [PERFORM] Performance issues
>
> On 17.3.2015 16:24, Thomas Kellerer wrote:
> > Tomas Vondra schrieb am 17.03.2015 um 15:43:
> >> On 17.3.2015 15:19, Thomas Kellerer wrote:
> >>> Tomas Vondra schrieb am 17.03.2015 um 14:55:
> >>>> (2) using window functions, e.g. like this:
> >>>>
> >>>> SELECT * FROM (
> >>>> SELECT *,
> >>>> ROW_NUMBER() OVER (PARTITION BY touchpoint_execution_id
> >>>> ORDER BY FROM max_creation_dt) AS rn
> >>>> FROM s_f_touchpoint_execution_status_history
> >>>> ) foo WHERE rn = 1
> >>>>
> >>>> But estimating this is also rather difficult ...
> >>>
> >>>
> >>> From my experience rewriting something like the above using DISTINCT
> >>> ON is usually faster.
> >>
> >> How do you get the last record (with respect to a timestamp column)
> >> using a DISTINCT ON?
> >
> > You need to use "order by ... desc". See here:
> > http://sqlfiddle.com/#!15/d4846/2
>
> Nice, thanks!
>
> >
> > Btw: your row_number() usage wouldn't return the "latest" row either.
> > It would return the "oldest" row.
>
> Oh, right. I forgot the DESC in the window.
>
>
> --
> Tomas Vondra http://www.2ndQuadrant.com/
> PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
>
>
> --
> Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-performance
>
>
> --
> Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-performance
>
"But this is not a production-suitable setting. So what do you think how to
get a work around this?"
What about creating a read-only replica and apply this setting there?
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Performance issues
2015-03-13 11:44 Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 12:58 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 13:03 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 13:10 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 19:59 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 20:36 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-13 20:46 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 22:03 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 22:25 ` Re: Performance issues Scott Marlowe <scott.marlowe@gmail.com>
2015-03-13 23:28 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-14 00:12 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-16 19:43 ` Re: Performance issues Jim Nasby <Jim.Nasby@BlueTreble.com>
2015-03-16 20:59 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 00:06 ` Re: Performance issues Jim Nasby <Jim.Nasby@BlueTreble.com>
2015-03-17 11:07 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-17 11:45 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 12:10 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-17 13:55 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 14:19 ` Re: Performance issues Thomas Kellerer <spam_eater@gmx.net>
2015-03-17 14:43 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 15:24 ` Re: Performance issues Thomas Kellerer <spam_eater@gmx.net>
2015-03-18 17:31 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
@ 2015-03-18 18:36 ` Jerry Sievers <gsievers19@comcast.net>
2015-03-18 18:48 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2 siblings, 1 reply; 59+ messages in thread
From: Jerry Sievers @ 2015-03-18 18:36 UTC (permalink / raw)
To: vjoshi@zetainteractive.com; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; pgsql-performance
Vivekanand Joshi <vjoshi@zetainteractive.com> writes:
> So, here is the first taste of success and which gives me the confidence
> that if properly worked out with a good hardware and proper tuning,
> PostgreSQL could be a good replacement.
>
> Out of the 9 reports which needs to be migrated in PostgreSQL, 3 are now
> running.
>
> Report 4 was giving an issue and I will see it tomorrow.
>
> Just to inform you guys that, the thing that helped most is setting
> enable_nestloops to false worked. Plans are now not miscalculated.
>
> But this is not a production-suitable setting. So what do you think how to
> get a work around this?
Consider just disabling that setting for 1 or a few odd queries you have
for which they are known to plan badly.
begin;
set local enable_nestloops to false;
select ...;
commit/abort;
I'd say never make that sort of setting DB or cluster-wide.
>
>
> Regards,
> Vivek
>
> -----Original Message-----
> From: pgsql-performance-owner@postgresql.org
> [mailto:pgsql-performance-owner@postgresql.org] On Behalf Of Tomas Vondra
> Sent: Tuesday, March 17, 2015 9:00 PM
> To: pgsql-performance@postgresql.org
> Subject: Re: [PERFORM] Performance issues
>
> On 17.3.2015 16:24, Thomas Kellerer wrote:
>> Tomas Vondra schrieb am 17.03.2015 um 15:43:
>>> On 17.3.2015 15:19, Thomas Kellerer wrote:
>>>> Tomas Vondra schrieb am 17.03.2015 um 14:55:
>>>>> (2) using window functions, e.g. like this:
>>>>>
>>>>> SELECT * FROM (
>>>>> SELECT *,
>>>>> ROW_NUMBER() OVER (PARTITION BY touchpoint_execution_id
>>>>> ORDER BY FROM max_creation_dt) AS rn
>>>>> FROM s_f_touchpoint_execution_status_history
>>>>> ) foo WHERE rn = 1
>>>>>
>>>>> But estimating this is also rather difficult ...
>>>>
>>>>
>>>> From my experience rewriting something like the above using DISTINCT
>>>> ON is usually faster.
>>>
>>> How do you get the last record (with respect to a timestamp column)
>>> using a DISTINCT ON?
>>
>> You need to use "order by ... desc". See here:
>> http://sqlfiddle.com/#!15/d4846/2
>
> Nice, thanks!
>
>>
>> Btw: your row_number() usage wouldn't return the "latest" row either.
>> It would return the "oldest" row.
>
> Oh, right. I forgot the DESC in the window.
>
>
> --
> Tomas Vondra http://www.2ndQuadrant.com/
> PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
>
>
> --
> Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-performance
--
Jerry Sievers
Postgres DBA/Development Consulting
e: postgres.consulting@comcast.net
p: 312.241.7800
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Performance issues
2015-03-13 11:44 Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 12:58 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 13:03 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 13:10 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 19:59 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 20:36 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-13 20:46 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 22:03 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 22:25 ` Re: Performance issues Scott Marlowe <scott.marlowe@gmail.com>
2015-03-13 23:28 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-14 00:12 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-16 19:43 ` Re: Performance issues Jim Nasby <Jim.Nasby@BlueTreble.com>
2015-03-16 20:59 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 00:06 ` Re: Performance issues Jim Nasby <Jim.Nasby@BlueTreble.com>
2015-03-17 11:07 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-17 11:45 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 12:10 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-17 13:55 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 14:19 ` Re: Performance issues Thomas Kellerer <spam_eater@gmx.net>
2015-03-17 14:43 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 15:24 ` Re: Performance issues Thomas Kellerer <spam_eater@gmx.net>
2015-03-18 17:31 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-18 18:36 ` Re: Performance issues Jerry Sievers <gsievers19@comcast.net>
@ 2015-03-18 18:48 ` Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-18 19:07 ` Re: Performance issues Vitalii Tymchyshyn <vit@tym.im>
2015-03-21 07:39 ` Re: Performance issues Josh Krupka <jkrupka@gmail.com>
0 siblings, 2 replies; 59+ messages in thread
From: Vivekanand Joshi @ 2015-03-18 18:48 UTC (permalink / raw)
To: Jerry Sievers <gsievers19@comcast.net>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; pgsql-performance
The issue here is that the queries are running inside a Jasper Reports. So
we cannot set this only for a one single query.
We are accessing our reports from a web-browser, which in turn runs the
report from Application Server (Jasper). This server connects to
PostgreSQL server.
Inside a JRXML(Jasper report file) file we cannot set this parameter.
I am attaching a JRXML file for a feel. You can open this file in
notepad. I don't think we can set server level property in this file. So
how about a workaround?
Vivek
-----Original Message-----
From: Jerry Sievers [mailto:gsievers19@comcast.net]
Sent: Thursday, March 19, 2015 12:06 AM
To: vjoshi@zetainteractive.com
Cc: Tomas Vondra; pgsql-performance@postgresql.org
Subject: Re: [PERFORM] Performance issues
Vivekanand Joshi <vjoshi@zetainteractive.com> writes:
> So, here is the first taste of success and which gives me the
> confidence that if properly worked out with a good hardware and proper
> tuning, PostgreSQL could be a good replacement.
>
> Out of the 9 reports which needs to be migrated in PostgreSQL, 3 are
> now running.
>
> Report 4 was giving an issue and I will see it tomorrow.
>
> Just to inform you guys that, the thing that helped most is setting
> enable_nestloops to false worked. Plans are now not miscalculated.
>
> But this is not a production-suitable setting. So what do you think
> how to get a work around this?
Consider just disabling that setting for 1 or a few odd queries you have
for which they are known to plan badly.
begin;
set local enable_nestloops to false;
select ...;
commit/abort;
I'd say never make that sort of setting DB or cluster-wide.
>
>
> Regards,
> Vivek
>
> -----Original Message-----
> From: pgsql-performance-owner@postgresql.org
> [mailto:pgsql-performance-owner@postgresql.org] On Behalf Of Tomas
> Vondra
> Sent: Tuesday, March 17, 2015 9:00 PM
> To: pgsql-performance@postgresql.org
> Subject: Re: [PERFORM] Performance issues
>
> On 17.3.2015 16:24, Thomas Kellerer wrote:
>> Tomas Vondra schrieb am 17.03.2015 um 15:43:
>>> On 17.3.2015 15:19, Thomas Kellerer wrote:
>>>> Tomas Vondra schrieb am 17.03.2015 um 14:55:
>>>>> (2) using window functions, e.g. like this:
>>>>>
>>>>> SELECT * FROM (
>>>>> SELECT *,
>>>>> ROW_NUMBER() OVER (PARTITION BY touchpoint_execution_id
>>>>> ORDER BY FROM max_creation_dt) AS rn
>>>>> FROM s_f_touchpoint_execution_status_history
>>>>> ) foo WHERE rn = 1
>>>>>
>>>>> But estimating this is also rather difficult ...
>>>>
>>>>
>>>> From my experience rewriting something like the above using
>>>> DISTINCT ON is usually faster.
>>>
>>> How do you get the last record (with respect to a timestamp column)
>>> using a DISTINCT ON?
>>
>> You need to use "order by ... desc". See here:
>> http://sqlfiddle.com/#!15/d4846/2
>
> Nice, thanks!
>
>>
>> Btw: your row_number() usage wouldn't return the "latest" row either.
>> It would return the "oldest" row.
>
> Oh, right. I forgot the DESC in the window.
>
>
> --
> Tomas Vondra http://www.2ndQuadrant.com/
> PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
>
>
> --
> Sent via pgsql-performance mailing list
> (pgsql-performance@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-performance
--
Jerry Sievers
Postgres DBA/Development Consulting
e: postgres.consulting@comcast.net
p: 312.241.7800
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
Attachments:
AvgEmailsRecieved.jrxml (application/octet-stream, 8.1K) download
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Performance issues
2015-03-13 11:44 Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 12:58 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 13:03 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 13:10 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 19:59 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 20:36 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-13 20:46 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 22:03 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 22:25 ` Re: Performance issues Scott Marlowe <scott.marlowe@gmail.com>
2015-03-13 23:28 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-14 00:12 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-16 19:43 ` Re: Performance issues Jim Nasby <Jim.Nasby@BlueTreble.com>
2015-03-16 20:59 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 00:06 ` Re: Performance issues Jim Nasby <Jim.Nasby@BlueTreble.com>
2015-03-17 11:07 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-17 11:45 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 12:10 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-17 13:55 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 14:19 ` Re: Performance issues Thomas Kellerer <spam_eater@gmx.net>
2015-03-17 14:43 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 15:24 ` Re: Performance issues Thomas Kellerer <spam_eater@gmx.net>
2015-03-18 17:31 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-18 18:36 ` Re: Performance issues Jerry Sievers <gsievers19@comcast.net>
2015-03-18 18:48 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
@ 2015-03-18 19:07 ` Vitalii Tymchyshyn <vit@tym.im>
1 sibling, 0 replies; 59+ messages in thread
From: Vitalii Tymchyshyn @ 2015-03-18 19:07 UTC (permalink / raw)
To: vjoshi@zetainteractive.com, Jerry Sievers <gsievers19@comcast.net>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; pgsql-performance
You can set it for the db user or use stored proc.
Best regards, Vitalii Tymchyshyn
Ср, 18 бер. 2015 14:48 Vivekanand Joshi <vjoshi@zetainteractive.com> пише:
> The issue here is that the queries are running inside a Jasper Reports. So
> we cannot set this only for a one single query.
>
> We are accessing our reports from a web-browser, which in turn runs the
> report from Application Server (Jasper). This server connects to
> PostgreSQL server.
>
> Inside a JRXML(Jasper report file) file we cannot set this parameter.
>
> I am attaching a JRXML file for a feel. You can open this file in
> notepad. I don't think we can set server level property in this file. So
> how about a workaround?
>
> Vivek
>
>
>
> -----Original Message-----
> From: Jerry Sievers [mailto:gsievers19@comcast.net]
> Sent: Thursday, March 19, 2015 12:06 AM
> To: vjoshi@zetainteractive.com
> Cc: Tomas Vondra; pgsql-performance@postgresql.org
> Subject: Re: [PERFORM] Performance issues
>
> Vivekanand Joshi <vjoshi@zetainteractive.com> writes:
>
> > So, here is the first taste of success and which gives me the
> > confidence that if properly worked out with a good hardware and proper
> > tuning, PostgreSQL could be a good replacement.
> >
> > Out of the 9 reports which needs to be migrated in PostgreSQL, 3 are
> > now running.
> >
> > Report 4 was giving an issue and I will see it tomorrow.
> >
> > Just to inform you guys that, the thing that helped most is setting
> > enable_nestloops to false worked. Plans are now not miscalculated.
> >
> > But this is not a production-suitable setting. So what do you think
> > how to get a work around this?
>
> Consider just disabling that setting for 1 or a few odd queries you have
> for which they are known to plan badly.
>
> begin;
> set local enable_nestloops to false;
> select ...;
> commit/abort;
>
> I'd say never make that sort of setting DB or cluster-wide.
>
>
> >
> >
> > Regards,
> > Vivek
> >
> > -----Original Message-----
> > From: pgsql-performance-owner@postgresql.org
> > [mailto:pgsql-performance-owner@postgresql.org] On Behalf Of Tomas
> > Vondra
> > Sent: Tuesday, March 17, 2015 9:00 PM
> > To: pgsql-performance@postgresql.org
> > Subject: Re: [PERFORM] Performance issues
> >
> > On 17.3.2015 16:24, Thomas Kellerer wrote:
> >> Tomas Vondra schrieb am 17.03.2015 um 15:43:
> >>> On 17.3.2015 15:19, Thomas Kellerer wrote:
> >>>> Tomas Vondra schrieb am 17.03.2015 um 14:55:
> >>>>> (2) using window functions, e.g. like this:
> >>>>>
> >>>>> SELECT * FROM (
> >>>>> SELECT *,
> >>>>> ROW_NUMBER() OVER (PARTITION BY touchpoint_execution_id
> >>>>> ORDER BY FROM max_creation_dt) AS rn
> >>>>> FROM s_f_touchpoint_execution_status_history
> >>>>> ) foo WHERE rn = 1
> >>>>>
> >>>>> But estimating this is also rather difficult ...
> >>>>
> >>>>
> >>>> From my experience rewriting something like the above using
> >>>> DISTINCT ON is usually faster.
> >>>
> >>> How do you get the last record (with respect to a timestamp column)
> >>> using a DISTINCT ON?
> >>
> >> You need to use "order by ... desc". See here:
> >> http://sqlfiddle.com/#!15/d4846/2
> >
> > Nice, thanks!
> >
> >>
> >> Btw: your row_number() usage wouldn't return the "latest" row either.
> >> It would return the "oldest" row.
> >
> > Oh, right. I forgot the DESC in the window.
> >
> >
> > --
> > Tomas Vondra http://www.2ndQuadrant.com/
> > PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
> >
> >
> > --
> > Sent via pgsql-performance mailing list
> > (pgsql-performance@postgresql.org)
> > To make changes to your subscription:
> > http://www.postgresql.org/mailpref/pgsql-performance
>
> --
> Jerry Sievers
> Postgres DBA/Development Consulting
> e: postgres.consulting@comcast.net
> p: 312.241.7800
>
> --
> Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-performance
>
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Performance issues
2015-03-13 11:44 Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 12:58 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 13:03 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 13:10 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 19:59 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 20:36 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-13 20:46 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 22:03 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 22:25 ` Re: Performance issues Scott Marlowe <scott.marlowe@gmail.com>
2015-03-13 23:28 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-14 00:12 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-16 19:43 ` Re: Performance issues Jim Nasby <Jim.Nasby@BlueTreble.com>
2015-03-16 20:59 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 00:06 ` Re: Performance issues Jim Nasby <Jim.Nasby@BlueTreble.com>
2015-03-17 11:07 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-17 11:45 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 12:10 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-17 13:55 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 14:19 ` Re: Performance issues Thomas Kellerer <spam_eater@gmx.net>
2015-03-17 14:43 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 15:24 ` Re: Performance issues Thomas Kellerer <spam_eater@gmx.net>
2015-03-18 17:31 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-18 18:36 ` Re: Performance issues Jerry Sievers <gsievers19@comcast.net>
2015-03-18 18:48 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
@ 2015-03-21 07:39 ` Josh Krupka <jkrupka@gmail.com>
2015-03-22 21:50 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
1 sibling, 1 reply; 59+ messages in thread
From: Josh Krupka @ 2015-03-21 07:39 UTC (permalink / raw)
To: vjoshi@zetainteractive.com; +Cc: pgsql-performance
The other approaches of fixing the estimates, cost params, etc are the
right way of fixing it. *However* if you needed a quick fix for just this
report and can't find a way of setting it in Jaspersoft for just the report
(I don't think it will let you run multiple sql statements by default,
maybe not at all) there are still a couple more options. You can define a
new datasource in jasper, point this report to that datasource, and have
that new datasource configured to not use the nested loops. You could do
that either by making the new datasource use a different user than
everything else, and disable nested loops for that user in postgres, or you
could probably have the datasource initialization process disable nested
loops.
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Performance issues
2015-03-13 11:44 Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 12:58 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 13:03 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 13:10 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 19:59 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 20:36 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-13 20:46 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 22:03 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 22:25 ` Re: Performance issues Scott Marlowe <scott.marlowe@gmail.com>
2015-03-13 23:28 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-14 00:12 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-16 19:43 ` Re: Performance issues Jim Nasby <Jim.Nasby@BlueTreble.com>
2015-03-16 20:59 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 00:06 ` Re: Performance issues Jim Nasby <Jim.Nasby@BlueTreble.com>
2015-03-17 11:07 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-17 11:45 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 12:10 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-17 13:55 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 14:19 ` Re: Performance issues Thomas Kellerer <spam_eater@gmx.net>
2015-03-17 14:43 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 15:24 ` Re: Performance issues Thomas Kellerer <spam_eater@gmx.net>
2015-03-18 17:31 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-18 18:36 ` Re: Performance issues Jerry Sievers <gsievers19@comcast.net>
2015-03-18 18:48 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-21 07:39 ` Re: Performance issues Josh Krupka <jkrupka@gmail.com>
@ 2015-03-22 21:50 ` Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-22 22:01 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 1 reply; 59+ messages in thread
From: Vivekanand Joshi @ 2015-03-22 21:50 UTC (permalink / raw)
To: Josh Krupka <jkrupka@gmail.com>; +Cc: pgsql-performance
Any documentation regarding how to configure postgresql.conf file as per
individual user?
On 21 Mar 2015 13:10, "Josh Krupka" <jkrupka@gmail.com> wrote:
> The other approaches of fixing the estimates, cost params, etc are the
> right way of fixing it. *However* if you needed a quick fix for just this
> report and can't find a way of setting it in Jaspersoft for just the report
> (I don't think it will let you run multiple sql statements by default,
> maybe not at all) there are still a couple more options. You can define a
> new datasource in jasper, point this report to that datasource, and have
> that new datasource configured to not use the nested loops. You could do
> that either by making the new datasource use a different user than
> everything else, and disable nested loops for that user in postgres, or you
> could probably have the datasource initialization process disable nested
> loops.
>
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Performance issues
2015-03-13 11:44 Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 12:58 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 13:03 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 13:10 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 19:59 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 20:36 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-13 20:46 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 22:03 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 22:25 ` Re: Performance issues Scott Marlowe <scott.marlowe@gmail.com>
2015-03-13 23:28 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-14 00:12 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-16 19:43 ` Re: Performance issues Jim Nasby <Jim.Nasby@BlueTreble.com>
2015-03-16 20:59 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 00:06 ` Re: Performance issues Jim Nasby <Jim.Nasby@BlueTreble.com>
2015-03-17 11:07 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-17 11:45 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 12:10 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-17 13:55 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 14:19 ` Re: Performance issues Thomas Kellerer <spam_eater@gmx.net>
2015-03-17 14:43 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 15:24 ` Re: Performance issues Thomas Kellerer <spam_eater@gmx.net>
2015-03-18 17:31 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-18 18:36 ` Re: Performance issues Jerry Sievers <gsievers19@comcast.net>
2015-03-18 18:48 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-21 07:39 ` Re: Performance issues Josh Krupka <jkrupka@gmail.com>
2015-03-22 21:50 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
@ 2015-03-22 22:01 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 0 replies; 59+ messages in thread
From: Tomas Vondra @ 2015-03-22 22:01 UTC (permalink / raw)
To: pgsql-performance
On 22.3.2015 22:50, Vivekanand Joshi wrote:
> Any documentation regarding how to configure postgresql.conf file as per
> individual user?
That can't be done in postgresql.conf, but by ALTER ROLE commands.
--
Tomas Vondra http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Performance issues
2015-03-13 11:44 Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 12:58 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 13:03 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 13:10 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 19:59 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 20:36 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-13 20:46 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 22:03 ` Re: Performance issues Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 22:25 ` Re: Performance issues Scott Marlowe <scott.marlowe@gmail.com>
2015-03-13 23:28 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-14 00:12 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-16 19:43 ` Re: Performance issues Jim Nasby <Jim.Nasby@BlueTreble.com>
2015-03-16 20:59 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 00:06 ` Re: Performance issues Jim Nasby <Jim.Nasby@BlueTreble.com>
2015-03-17 11:07 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-17 11:45 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 12:10 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-17 13:55 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 14:19 ` Re: Performance issues Thomas Kellerer <spam_eater@gmx.net>
2015-03-17 14:43 ` Re: Performance issues Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 15:24 ` Re: Performance issues Thomas Kellerer <spam_eater@gmx.net>
2015-03-18 17:31 ` Re: Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
@ 2015-03-18 19:23 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2 siblings, 0 replies; 59+ messages in thread
From: Tomas Vondra @ 2015-03-18 19:23 UTC (permalink / raw)
To: vjoshi@zetainteractive.com; pgsql-performance
Hi,
On 18.3.2015 18:31, Vivekanand Joshi wrote:
> So, here is the first taste of success and which gives me the
> confidence that if properly worked out with a good hardware and
> proper tuning, PostgreSQL could be a good replacement.
>
> Out of the 9 reports which needs to be migrated in PostgreSQL, 3 are
> now running.
>
> Report 4 was giving an issue and I will see it tomorrow.
>
> Just to inform you guys that, the thing that helped most is setting
> enable_nestloops to false worked. Plans are now not miscalculated.
The estimates are still miscalculated, but you're forcing the database
not to use the nested loop. The problem is the nested loop may be
appropriate in some cases (maybe only in a few places of the plan) so
this is really corse-grained solution.
> But this is not a production-suitable setting. So what do you think
> how to get a work around this?
(a) Try to identify why the queries are poorly estimated, and rephrase
them somehow. This is the best solution, but takes time, expertise
and may not be feasible in some cases.
(b) Tweak the database structure, possibly introducing intermediate
tables, materialized views (or tables maintained by triggers - this
might work for the 'latest record' subquery), etc.
(c) Try to tweak the cost parameters, to make the nested loops more
expensive (and thus less likely to be selected), but in a more
gradual way than enable_nestloops=false.
regards
--
Tomas Vondra http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 59+ messages in thread
end of thread, other threads:[~2015-03-22 22:01 UTC | newest]
Thread overview: 59+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2011-03-07 13:45 Performance issues Andreas Forø Tollefsen <andreasft@gmail.com>
2011-03-07 14:01 ` Kenneth Marshall <ktm@rice.edu>
2011-03-07 14:17 ` Andreas Forø Tollefsen <andreasft@gmail.com>
2011-03-07 14:22 ` Kenneth Marshall <ktm@rice.edu>
2011-03-07 14:29 ` Andreas Forø Tollefsen <andreasft@gmail.com>
2011-03-07 15:34 ` Oleg Bartunov <oleg@sai.msu.su>
2011-03-07 21:49 ` Andreas Forø Tollefsen <andreasft@gmail.com>
2011-03-07 23:29 ` David Kerr <dmk@mr-paradox.net>
2011-03-08 00:38 ` Tom Lane <tgl@sss.pgh.pa.us>
2011-03-08 08:42 ` Andreas Forø Tollefsen <andreasft@gmail.com>
2011-03-08 15:21 ` Andy Colson <andy@squeakycode.net>
2011-03-08 16:58 ` Andreas Forø Tollefsen <andreasft@gmail.com>
2011-03-08 17:00 ` Andreas Forø Tollefsen <andreasft@gmail.com>
2011-03-08 19:13 ` Andy Colson <andy@squeakycode.net>
2015-03-13 11:44 Performance issues Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 12:58 ` Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 13:03 ` Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 13:10 ` Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 19:59 ` Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 20:36 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-13 20:46 ` Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-13 21:20 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-13 22:03 ` Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-13 22:25 ` Scott Marlowe <scott.marlowe@gmail.com>
2015-03-13 23:28 ` Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-14 00:12 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-14 21:06 ` Gavin Flower <GavinFlower@archidevsys.co.nz>
2015-03-14 21:23 ` Varadharajan Mukundan <srinathsmn@gmail.com>
2015-03-14 21:32 ` Gavin Flower <GavinFlower@archidevsys.co.nz>
2015-03-16 12:24 ` Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-16 17:49 ` Marc Mamin <M.Mamin@intershop.de>
2015-03-16 18:24 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-16 18:32 ` Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-16 19:43 ` Jim Nasby <Jim.Nasby@BlueTreble.com>
2015-03-16 20:59 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 00:06 ` Jim Nasby <Jim.Nasby@BlueTreble.com>
2015-03-17 07:41 ` Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-17 11:42 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 11:07 ` Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-17 11:45 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 12:05 ` Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-17 12:10 ` Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-17 13:55 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 14:19 ` Thomas Kellerer <spam_eater@gmx.net>
2015-03-17 14:43 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 15:10 ` Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-17 15:28 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-17 15:16 ` Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-17 15:24 ` Thomas Kellerer <spam_eater@gmx.net>
2015-03-17 15:30 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-18 17:31 ` Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-18 17:51 ` Felipe Santos <felipepts@gmail.com>
2015-03-18 18:36 ` Jerry Sievers <gsievers19@comcast.net>
2015-03-18 18:48 ` Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-18 19:07 ` Vitalii Tymchyshyn <vit@tym.im>
2015-03-21 07:39 ` Josh Krupka <jkrupka@gmail.com>
2015-03-22 21:50 ` Vivekanand Joshi <vjoshi@zetainteractive.com>
2015-03-22 22:01 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-18 19:23 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox