pg.ddx.io pgsql-performance@postgresql.org mailing list archive
help / color / mirror / Atom feedtwo table join just not fast enough.
4+ messages / 3 participants
[nested] [flat]
* two table join just not fast enough.
@ 2011-11-02 20:12 Brian Fehrle <brianf@consistentstate.com>
2011-11-02 23:53 ` Re: two table join just not fast enough. Tom Lane <tgl@sss.pgh.pa.us>
2011-11-03 01:47 ` Re: two table join just not fast enough. Mark Kirkwood <mark.kirkwood@catalyst.net.nz>
0 siblings, 2 replies; 4+ messages in thread
From: Brian Fehrle @ 2011-11-02 20:12 UTC (permalink / raw)
To: pgsql-performance
Hi all,
I've got a query that I need to squeeze as much speed out of as I can.
When I execute this query, the average time it takes is about 190 ms. I
increased my work_mem from 1 MB to 50MB and it decreased the timing down
to an average of 170 ms, but that's still not fast enough. This query is
executed extremely frequently, so much of it should be easily cached.
Some settings
work_mem = 50MB
shared_buffers = 5GB
I've made sure that the two tables are recently analyzed (with
default_stats to 100, 400, and 1500 even), and table bloat is low (150
meg table has 7 megs bloat).
Here's the query:
SELECT yankee.my_id
FROM yankee
INNER JOIN hotel_zulu
ON hotel_zulu.my_id = yankee.zulu_id
AND hotel_zulu.type IN ('string1', 'string2', 'string3', 'string4')
WHERE yankee.your_id=402513;
And here is a query plan.
Hash Join (cost=17516.470..26386.660 rows=27624 width=4) (actual time=309.194..395.135 rows=12384 loops=1)
Hash Cond: (yankee.alpha = hotel_zulu.quebec)
-> Bitmap Heap Scan on yankee (cost=1066.470..8605.770 rows=27624 width=20) (actual time=5.178..34.693 rows=26963 loops=1)
Recheck Cond: (mike = 402513)
-> Bitmap Index Scan on hotel_alpha (cost=0.000..1059.570 rows=27624 width=0) (actual time=4.770..4.770 rows=26967 loops=1)
Index Cond: (mike = 402513)
-> Hash (cost=14465.080..14465.080 rows=114154 width=16) (actual time=303.717..303.717 rows=129908 loops=1)
Buckets: 4096 Batches: 8 Memory Usage: 784kB
-> Bitmap Heap Scan on hotel_zulu (cost=2461.770..14465.080 rows=114154 width=16) (actual time=25.642..185.253 rows=129908 loops=1)
Recheck Cond: ((two)::golf = ANY ('xray'::golf[]))
-> Bitmap Index Scan on kilo (cost=0.000..2433.230 rows=114154 width=0) (actual time=23.887..23.887 rows=130292 loops=1)
Index Cond: ((two)::golf = ANY ('xray'::golf[]))
One thing I notice is the rows estimated is 27624 and the actual rows
returned is 12384. Quite a bit different.
Table 2 (known here as hotel_zulu) is being joined on zulu_id to the
first table, and then a where clause on the column 'type'. There are
single column indexes on each of these columns, and any multi column
index I put on these are just ignored by the planner.
Any thoughts on ways to tweak this?
- Brian F
^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: two table join just not fast enough.
2011-11-02 20:12 two table join just not fast enough. Brian Fehrle <brianf@consistentstate.com>
@ 2011-11-02 23:53 ` Tom Lane <tgl@sss.pgh.pa.us>
2011-11-03 00:33 ` Re: two table join just not fast enough. Brian Fehrle <brianf@consistentstate.com>
1 sibling, 1 reply; 4+ messages in thread
From: Tom Lane @ 2011-11-02 23:53 UTC (permalink / raw)
To: Brian Fehrle <brianf@consistentstate.com>; +Cc: pgsql-performance
Brian Fehrle <brianf@consistentstate.com> writes:
> I've got a query that I need to squeeze as much speed out of as I can.
Hmm ... are you really sure this is being run with work_mem = 50MB?
The hash join is getting "batched", which means the executor thinks it's
working under a memory constraint significantly less than the size of
the filtered inner relation, which should be no more than a couple
megabytes according to this.
I'm not sure how much that will save, since the hashjoin seems to be
reasonably speedy anyway, but there's not much other fat to trim here.
One minor suggestion is to think whether you really need string
comparisons here or could convert that to use of an enum type.
String compares ain't cheap, especially not in non-C locales.
regards, tom lane
^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: two table join just not fast enough.
2011-11-02 20:12 two table join just not fast enough. Brian Fehrle <brianf@consistentstate.com>
2011-11-02 23:53 ` Re: two table join just not fast enough. Tom Lane <tgl@sss.pgh.pa.us>
@ 2011-11-03 00:33 ` Brian Fehrle <brianf@consistentstate.com>
0 siblings, 0 replies; 4+ messages in thread
From: Brian Fehrle @ 2011-11-03 00:33 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: pgsql-performance
Thanks Tom,
And looks like I pasted an older explain plan, which is almost exactly
the same as the one with 50MB work_mem, except for the hash join
'buckets' part which used more memory and only one 'bucket' so to speak.
When running with the 50MB work_mem over 1MB work_mem, the query went
from an average of 190 ms to 169 ms, so it did help some but it wasn't a
game changer (I even found for this specific query, 6MB of work_mem was
the most that would actually help me).
I have other plans to try to get this thing running faster, I'll be
exploring them tomorrow, as well as looking at using an enum type.
- Brian F
On 11/02/2011 05:53 PM, Tom Lane wrote:
> Brian Fehrle<brianf@consistentstate.com> writes:
>> I've got a query that I need to squeeze as much speed out of as I can.
> Hmm ... are you really sure this is being run with work_mem = 50MB?
> The hash join is getting "batched", which means the executor thinks it's
> working under a memory constraint significantly less than the size of
> the filtered inner relation, which should be no more than a couple
> megabytes according to this.
>
> I'm not sure how much that will save, since the hashjoin seems to be
> reasonably speedy anyway, but there's not much other fat to trim here.
>
> One minor suggestion is to think whether you really need string
> comparisons here or could convert that to use of an enum type.
> String compares ain't cheap, especially not in non-C locales.
>
> regards, tom lane
^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: two table join just not fast enough.
2011-11-02 20:12 two table join just not fast enough. Brian Fehrle <brianf@consistentstate.com>
@ 2011-11-03 01:47 ` Mark Kirkwood <mark.kirkwood@catalyst.net.nz>
1 sibling, 0 replies; 4+ messages in thread
From: Mark Kirkwood @ 2011-11-03 01:47 UTC (permalink / raw)
To: Brian Fehrle <brianf@consistentstate.com>; +Cc: pgsql-performance
On 03/11/11 09:12, Brian Fehrle wrote:
>
>
> And here is a query plan.
>
> Hash Join (cost=17516.470..26386.660 rows=27624 width=4) (actual
> time=309.194..395.135 rows=12384 loops=1)
> Hash Cond: (yankee.alpha = hotel_zulu.quebec)
> -> Bitmap Heap Scan on yankee (cost=1066.470..8605.770 rows=27624
> width=20) (actual time=5.178..34.693 rows=26963 loops=1)
> Recheck Cond: (mike = 402513)
> -> Bitmap Index Scan on hotel_alpha (cost=0.000..1059.570
> rows=27624 width=0) (actual time=4.770..4.770 rows=26967 loops=1)
> Index Cond: (mike = 402513)
> -> Hash (cost=14465.080..14465.080 rows=114154 width=16) (actual
> time=303.717..303.717 rows=129908 loops=1)
> Buckets: 4096 Batches: 8 Memory Usage: 784kB
> -> Bitmap Heap Scan on hotel_zulu (cost=2461.770..14465.080
> rows=114154 width=16) (actual time=25.642..185.253 rows=129908 loops=1)
> Recheck Cond: ((two)::golf = ANY ('xray'::golf[]))
> -> Bitmap Index Scan on kilo (cost=0.000..2433.230
> rows=114154 width=0) (actual time=23.887..23.887 rows=130292 loops=1)
> Index Cond: ((two)::golf = ANY ('xray'::golf[]))
>
Might be worth posting table definitions, as this plan does not
immediately look like it came from the query you posted. Also unless I
am misreading the output looks like you have some custom datatypes (e.g
'golf'), so more info there could be useful too.
When we have that, there may be something to be learned from examining
the pg_stats data for the join and predicate columns used in these queries.
regards
Mark
^ permalink raw reply [nested|flat] 4+ messages in thread
end of thread, other threads:[~2011-11-03 01:47 UTC | newest]
Thread overview: 4+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2011-11-02 20:12 two table join just not fast enough. Brian Fehrle <brianf@consistentstate.com>
2011-11-02 23:53 ` Tom Lane <tgl@sss.pgh.pa.us>
2011-11-03 00:33 ` Brian Fehrle <brianf@consistentstate.com>
2011-11-03 01:47 ` Mark Kirkwood <mark.kirkwood@catalyst.net.nz>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox