public inbox for [email protected]
help / color / mirror / Atom feedAsymmetric partition-wise JOIN
28+ messages / 14 participants
[nested] [flat]
* Asymmetric partition-wise JOIN
@ 2019-08-12 06:03 Kohei KaiGai <[email protected]>
2019-08-22 16:05 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
0 siblings, 1 reply; 28+ messages in thread
From: Kohei KaiGai @ 2019-08-12 06:03 UTC (permalink / raw)
To: pgsql-hackers
Hello,
PostgreSQL optimizer right now considers join pairs on only
non-partition - non-partition or
partition-leaf - partition-leaf relations. On the other hands, it is
harmless and makes sense to
consider a join pair on non-partition - partition-leaf.
See the example below. ptable is partitioned by hash, and contains 10M
rows. ftable is not
partitioned and contains 50 rows. Most of ptable::fkey shall not have
matched rows in this
join.
create table ptable (fkey int, dist text) partition by hash (dist);
create table ptable_p0 partition of ptable for values with (modulus 3,
remainder 0);
create table ptable_p1 partition of ptable for values with (modulus 3,
remainder 1);
create table ptable_p2 partition of ptable for values with (modulus 3,
remainder 2);
insert into ptable (select x % 10000, md5(x::text) from
generate_series(1,10000000) x);
create table ftable (pkey int primary key, memo text);
insert into ftable (select x, 'ftable__#' || x::text from
generate_series(1,50) x);
vacuum analyze;
postgres=# explain analyze select count(*) from ptable p, ftable f
where p.fkey = f.pkey;
QUERY PLAN
-------------------------------------------------------------------------------------------------------------------------------------------
Aggregate (cost=266393.38..266393.39 rows=1 width=8) (actual
time=2333.193..2333.194 rows=1 loops=1)
-> Hash Join (cost=2.12..260143.38 rows=2500000 width=0) (actual
time=0.056..2330.079 rows=50000 loops=1)
Hash Cond: (p.fkey = f.pkey)
-> Append (cost=0.00..233335.00 rows=10000000 width=4)
(actual time=0.012..1617.268 rows=10000000 loops=1)
-> Seq Scan on ptable_p0 p (cost=0.00..61101.96
rows=3332796 width=4) (actual time=0.011..351.137 rows=3332796
loops=1)
-> Seq Scan on ptable_p1 p_1 (cost=0.00..61106.25
rows=3333025 width=4) (actual time=0.005..272.925 rows=3333025
loops=1)
-> Seq Scan on ptable_p2 p_2 (cost=0.00..61126.79
rows=3334179 width=4) (actual time=0.006..416.141 rows=3334179
loops=1)
-> Hash (cost=1.50..1.50 rows=50 width=4) (actual
time=0.033..0.034 rows=50 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 10kB
-> Seq Scan on ftable f (cost=0.00..1.50 rows=50
width=4) (actual time=0.004..0.017 rows=50 loops=1)
Planning Time: 0.286 ms
Execution Time: 2333.264 ms
(12 rows)
We can manually rewrite this query as follows:
postgres=# explain analyze select count(*) from (
select * from ptable_p0 p, ftable f where p.fkey =
f.pkey union all
select * from ptable_p1 p, ftable f where p.fkey =
f.pkey union all
select * from ptable_p2 p, ftable f where p.fkey = f.pkey) subqry;
Because Append does not process tuples that shall have no matched
tuples in ftable,
this query has cheaper cost and short query execution time.
(2333ms --> 1396ms)
postgres=# explain analyze select count(*) from (
select * from ptable_p0 p, ftable f where p.fkey =
f.pkey union all
select * from ptable_p1 p, ftable f where p.fkey =
f.pkey union all
select * from ptable_p2 p, ftable f where p.fkey = f.pkey) subqry;
QUERY PLAN
-------------------------------------------------------------------------------------------------------------------------------------------------
Aggregate (cost=210478.25..210478.26 rows=1 width=8) (actual
time=1396.024..1396.024 rows=1 loops=1)
-> Append (cost=2.12..210353.14 rows=50042 width=0) (actual
time=0.058..1393.008 rows=50000 loops=1)
-> Subquery Scan on "*SELECT* 1" (cost=2.12..70023.66
rows=16726 width=0) (actual time=0.057..573.197 rows=16789 loops=1)
-> Hash Join (cost=2.12..69856.40 rows=16726
width=72) (actual time=0.056..571.718 rows=16789 loops=1)
Hash Cond: (p.fkey = f.pkey)
-> Seq Scan on ptable_p0 p (cost=0.00..61101.96
rows=3332796 width=4) (actual time=0.009..255.791 rows=3332796
loops=1)
-> Hash (cost=1.50..1.50 rows=50 width=4)
(actual time=0.034..0.035 rows=50 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 10kB
-> Seq Scan on ftable f (cost=0.00..1.50
rows=50 width=4) (actual time=0.004..0.019 rows=50 loops=1)
-> Subquery Scan on "*SELECT* 2" (cost=2.12..70027.43
rows=16617 width=0) (actual time=0.036..409.712 rows=16578 loops=1)
-> Hash Join (cost=2.12..69861.26 rows=16617
width=72) (actual time=0.036..408.626 rows=16578 loops=1)
Hash Cond: (p_1.fkey = f_1.pkey)
-> Seq Scan on ptable_p1 p_1
(cost=0.00..61106.25 rows=3333025 width=4) (actual time=0.005..181.422
rows=3333025 loops=1)
-> Hash (cost=1.50..1.50 rows=50 width=4)
(actual time=0.020..0.020 rows=50 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 10kB
-> Seq Scan on ftable f_1
(cost=0.00..1.50 rows=50 width=4) (actual time=0.004..0.011 rows=50
loops=1)
-> Subquery Scan on "*SELECT* 3" (cost=2.12..70051.84
rows=16699 width=0) (actual time=0.025..407.103 rows=16633 loops=1)
-> Hash Join (cost=2.12..69884.85 rows=16699
width=72) (actual time=0.025..406.048 rows=16633 loops=1)
Hash Cond: (p_2.fkey = f_2.pkey)
-> Seq Scan on ptable_p2 p_2
(cost=0.00..61126.79 rows=3334179 width=4) (actual time=0.004..181.015
rows=3334179 loops=1)
-> Hash (cost=1.50..1.50 rows=50 width=4)
(actual time=0.014..0.014 rows=50 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 10kB
-> Seq Scan on ftable f_2
(cost=0.00..1.50 rows=50 width=4) (actual time=0.003..0.008 rows=50
loops=1)
Planning Time: 0.614 ms
Execution Time: 1396.131 ms
(25 rows)
How about your opinions for this kind of asymmetric partition-wise
JOIN support by the optimizer?
I think we can harmlessly push-down inner-join and left-join if
partition-leaf is left side.
Probably, we need to implement two key functionalities.
1. Construction of RelOpInfo for join on non-partition table and
partition-leafs for each pairs.
Instead of JoinPaths, this logic adds AppendPath that takes
asymmetric partition-wise join
paths as sub-paths. Other optimization logic is equivalent as we
are currently doing.
2. Allow to share the hash-table built from table scan distributed to
individual partition leafs.
In the above example, SeqScan on ftable and relevant Hash path
will make identical hash-
table for the upcoming hash-join. If sibling paths have equivalent
results, it is reasonable to
reuse it.
Best regards,
--
HeteroDB, Inc / The PG-Strom Project
KaiGai Kohei <[email protected]>
^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: Asymmetric partition-wise JOIN
2019-08-12 06:03 Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
@ 2019-08-22 16:05 ` Kohei KaiGai <[email protected]>
2019-08-23 22:01 ` Re: Asymmetric partition-wise JOIN Thomas Munro <[email protected]>
0 siblings, 1 reply; 28+ messages in thread
From: Kohei KaiGai @ 2019-08-22 16:05 UTC (permalink / raw)
To: pgsql-hackers
Hello,
Even though nobody has respond the thread, I tried to make a prototype of
the asymmetric partition-wise join support.
This feature tries to join non-partitioned and partitioned relation
before append.
See the example below:
create table ptable (dist int, a int, b int) partition by hash (dist);
create table ptable_p0 partition of ptable for values with (modulus 3,
remainder 0);
create table ptable_p1 partition of ptable for values with (modulus 3,
remainder 1);
create table ptable_p2 partition of ptable for values with (modulus 3,
remainder 2);
create table t1 (aid int, label text);
create table t2 (bid int, label text);
insert into ptable (select x, (1000*random())::int,
(1000*random())::int from generate_series(1,1000000) x);
insert into t1 (select x, md5(x::text) from generate_series(1,50) x);
insert into t2 (select x, md5(x::text) from generate_series(1,50) x);
vacuum analyze ptable;
vacuum analyze t1;
vacuum analyze t2;
ptable.a has values between 0 and 1000, and t1.aid has values between 1 and 50.
Therefore, tables join on ptable and t1 by a=aid can reduce almost 95% rows.
On the other hands, t1 is not partitioned and join-keys are not partition keys.
So, Append must process million rows first, then HashJoin processes
the rows read
from the partitioned table, and 95% of them are eventually dropped.
On the other words, 95% of jobs by Append are waste of time and CPU cycles.
postgres=# explain select * from ptable, t1 where a = aid;
QUERY PLAN
------------------------------------------------------------------------------
Hash Join (cost=2.12..24658.62 rows=49950 width=49)
Hash Cond: (ptable_p0.a = t1.aid)
-> Append (cost=0.00..20407.00 rows=1000000 width=12)
-> Seq Scan on ptable_p0 (cost=0.00..5134.63 rows=333263 width=12)
-> Seq Scan on ptable_p1 (cost=0.00..5137.97 rows=333497 width=12)
-> Seq Scan on ptable_p2 (cost=0.00..5134.40 rows=333240 width=12)
-> Hash (cost=1.50..1.50 rows=50 width=37)
-> Seq Scan on t1 (cost=0.00..1.50 rows=50 width=37)
(8 rows)
The asymmetric partitionwise join allows to join non-partitioned tables and
partitioned tables prior to Append.
postgres=# set enable_partitionwise_join = on;
SET
postgres=# explain select * from ptable, t1 where a = aid;
QUERY PLAN
------------------------------------------------------------------------------
Append (cost=2.12..19912.62 rows=49950 width=49)
-> Hash Join (cost=2.12..6552.96 rows=16647 width=49)
Hash Cond: (ptable_p0.a = t1.aid)
-> Seq Scan on ptable_p0 (cost=0.00..5134.63 rows=333263 width=12)
-> Hash (cost=1.50..1.50 rows=50 width=37)
-> Seq Scan on t1 (cost=0.00..1.50 rows=50 width=37)
-> Hash Join (cost=2.12..6557.29 rows=16658 width=49)
Hash Cond: (ptable_p1.a = t1.aid)
-> Seq Scan on ptable_p1 (cost=0.00..5137.97 rows=333497 width=12)
-> Hash (cost=1.50..1.50 rows=50 width=37)
-> Seq Scan on t1 (cost=0.00..1.50 rows=50 width=37)
-> Hash Join (cost=2.12..6552.62 rows=16645 width=49)
Hash Cond: (ptable_p2.a = t1.aid)
-> Seq Scan on ptable_p2 (cost=0.00..5134.40 rows=333240 width=12)
-> Hash (cost=1.50..1.50 rows=50 width=37)
-> Seq Scan on t1 (cost=0.00..1.50 rows=50 width=37)
(16 rows)
We can consider the table join ptable X t1 above is equivalent to:
(ptable_p0 + ptable_p1 + ptable_p2) X t1
= (ptable_p0 X t1) + (ptable_p1 X t1) + (ptable_p2 X t1)
It returns an equivalent result, however, rows are already reduced by HashJoin
in the individual leaf of Append, so CPU-cycles consumed by Append node can
be cheaper.
On the other hands, it has a downside because t1 must be read 3 times and
hash table also must be built 3 times. It increases the expected cost,
so planner
may not choose the asymmetric partition-wise join plan.
One idea I have is, sibling HashJoin shares a hash table that was built once
by any of the sibling Hash plan. Right now, it is not implemented yet.
How about your thought for this feature?
Best regards,
2019年8月12日(月) 15:03 Kohei KaiGai <[email protected]>:
>
> Hello,
>
> PostgreSQL optimizer right now considers join pairs on only
> non-partition - non-partition or
> partition-leaf - partition-leaf relations. On the other hands, it is
> harmless and makes sense to
> consider a join pair on non-partition - partition-leaf.
>
> See the example below. ptable is partitioned by hash, and contains 10M
> rows. ftable is not
> partitioned and contains 50 rows. Most of ptable::fkey shall not have
> matched rows in this
> join.
>
> create table ptable (fkey int, dist text) partition by hash (dist);
> create table ptable_p0 partition of ptable for values with (modulus 3,
> remainder 0);
> create table ptable_p1 partition of ptable for values with (modulus 3,
> remainder 1);
> create table ptable_p2 partition of ptable for values with (modulus 3,
> remainder 2);
> insert into ptable (select x % 10000, md5(x::text) from
> generate_series(1,10000000) x);
>
> create table ftable (pkey int primary key, memo text);
> insert into ftable (select x, 'ftable__#' || x::text from
> generate_series(1,50) x);
> vacuum analyze;
>
> postgres=# explain analyze select count(*) from ptable p, ftable f
> where p.fkey = f.pkey;
> QUERY PLAN
> -------------------------------------------------------------------------------------------------------------------------------------------
> Aggregate (cost=266393.38..266393.39 rows=1 width=8) (actual
> time=2333.193..2333.194 rows=1 loops=1)
> -> Hash Join (cost=2.12..260143.38 rows=2500000 width=0) (actual
> time=0.056..2330.079 rows=50000 loops=1)
> Hash Cond: (p.fkey = f.pkey)
> -> Append (cost=0.00..233335.00 rows=10000000 width=4)
> (actual time=0.012..1617.268 rows=10000000 loops=1)
> -> Seq Scan on ptable_p0 p (cost=0.00..61101.96
> rows=3332796 width=4) (actual time=0.011..351.137 rows=3332796
> loops=1)
> -> Seq Scan on ptable_p1 p_1 (cost=0.00..61106.25
> rows=3333025 width=4) (actual time=0.005..272.925 rows=3333025
> loops=1)
> -> Seq Scan on ptable_p2 p_2 (cost=0.00..61126.79
> rows=3334179 width=4) (actual time=0.006..416.141 rows=3334179
> loops=1)
> -> Hash (cost=1.50..1.50 rows=50 width=4) (actual
> time=0.033..0.034 rows=50 loops=1)
> Buckets: 1024 Batches: 1 Memory Usage: 10kB
> -> Seq Scan on ftable f (cost=0.00..1.50 rows=50
> width=4) (actual time=0.004..0.017 rows=50 loops=1)
> Planning Time: 0.286 ms
> Execution Time: 2333.264 ms
> (12 rows)
>
> We can manually rewrite this query as follows:
>
> postgres=# explain analyze select count(*) from (
> select * from ptable_p0 p, ftable f where p.fkey =
> f.pkey union all
> select * from ptable_p1 p, ftable f where p.fkey =
> f.pkey union all
> select * from ptable_p2 p, ftable f where p.fkey = f.pkey) subqry;
>
> Because Append does not process tuples that shall have no matched
> tuples in ftable,
> this query has cheaper cost and short query execution time.
> (2333ms --> 1396ms)
>
> postgres=# explain analyze select count(*) from (
> select * from ptable_p0 p, ftable f where p.fkey =
> f.pkey union all
> select * from ptable_p1 p, ftable f where p.fkey =
> f.pkey union all
> select * from ptable_p2 p, ftable f where p.fkey = f.pkey) subqry;
> QUERY PLAN
> -------------------------------------------------------------------------------------------------------------------------------------------------
> Aggregate (cost=210478.25..210478.26 rows=1 width=8) (actual
> time=1396.024..1396.024 rows=1 loops=1)
> -> Append (cost=2.12..210353.14 rows=50042 width=0) (actual
> time=0.058..1393.008 rows=50000 loops=1)
> -> Subquery Scan on "*SELECT* 1" (cost=2.12..70023.66
> rows=16726 width=0) (actual time=0.057..573.197 rows=16789 loops=1)
> -> Hash Join (cost=2.12..69856.40 rows=16726
> width=72) (actual time=0.056..571.718 rows=16789 loops=1)
> Hash Cond: (p.fkey = f.pkey)
> -> Seq Scan on ptable_p0 p (cost=0.00..61101.96
> rows=3332796 width=4) (actual time=0.009..255.791 rows=3332796
> loops=1)
> -> Hash (cost=1.50..1.50 rows=50 width=4)
> (actual time=0.034..0.035 rows=50 loops=1)
> Buckets: 1024 Batches: 1 Memory Usage: 10kB
> -> Seq Scan on ftable f (cost=0.00..1.50
> rows=50 width=4) (actual time=0.004..0.019 rows=50 loops=1)
> -> Subquery Scan on "*SELECT* 2" (cost=2.12..70027.43
> rows=16617 width=0) (actual time=0.036..409.712 rows=16578 loops=1)
> -> Hash Join (cost=2.12..69861.26 rows=16617
> width=72) (actual time=0.036..408.626 rows=16578 loops=1)
> Hash Cond: (p_1.fkey = f_1.pkey)
> -> Seq Scan on ptable_p1 p_1
> (cost=0.00..61106.25 rows=3333025 width=4) (actual time=0.005..181.422
> rows=3333025 loops=1)
> -> Hash (cost=1.50..1.50 rows=50 width=4)
> (actual time=0.020..0.020 rows=50 loops=1)
> Buckets: 1024 Batches: 1 Memory Usage: 10kB
> -> Seq Scan on ftable f_1
> (cost=0.00..1.50 rows=50 width=4) (actual time=0.004..0.011 rows=50
> loops=1)
> -> Subquery Scan on "*SELECT* 3" (cost=2.12..70051.84
> rows=16699 width=0) (actual time=0.025..407.103 rows=16633 loops=1)
> -> Hash Join (cost=2.12..69884.85 rows=16699
> width=72) (actual time=0.025..406.048 rows=16633 loops=1)
> Hash Cond: (p_2.fkey = f_2.pkey)
> -> Seq Scan on ptable_p2 p_2
> (cost=0.00..61126.79 rows=3334179 width=4) (actual time=0.004..181.015
> rows=3334179 loops=1)
> -> Hash (cost=1.50..1.50 rows=50 width=4)
> (actual time=0.014..0.014 rows=50 loops=1)
> Buckets: 1024 Batches: 1 Memory Usage: 10kB
> -> Seq Scan on ftable f_2
> (cost=0.00..1.50 rows=50 width=4) (actual time=0.003..0.008 rows=50
> loops=1)
> Planning Time: 0.614 ms
> Execution Time: 1396.131 ms
> (25 rows)
>
> How about your opinions for this kind of asymmetric partition-wise
> JOIN support by the optimizer?
> I think we can harmlessly push-down inner-join and left-join if
> partition-leaf is left side.
>
> Probably, we need to implement two key functionalities.
> 1. Construction of RelOpInfo for join on non-partition table and
> partition-leafs for each pairs.
> Instead of JoinPaths, this logic adds AppendPath that takes
> asymmetric partition-wise join
> paths as sub-paths. Other optimization logic is equivalent as we
> are currently doing.
> 2. Allow to share the hash-table built from table scan distributed to
> individual partition leafs.
> In the above example, SeqScan on ftable and relevant Hash path
> will make identical hash-
> table for the upcoming hash-join. If sibling paths have equivalent
> results, it is reasonable to
> reuse it.
>
> Best regards,
> --
> HeteroDB, Inc / The PG-Strom Project
> KaiGai Kohei <[email protected]>
--
HeteroDB, Inc / The PG-Strom Project
KaiGai Kohei <[email protected]>
Attachments:
[application/octet-stream] pgsql13-asymmetric-partitionwise-join.v1.patch (20.0K, ../../CAOP8fzY9p5==63EDdAb14p1=eUxtqvVHrCXtEVnpnyJ5Mzq96Q@mail.gmail.com/2-pgsql13-asymmetric-partitionwise-join.v1.patch)
download | inline diff:
src/backend/optimizer/path/allpaths.c | 9 +-
src/backend/optimizer/path/joinpath.c | 9 ++
src/backend/optimizer/path/joinrels.c | 129 ++++++++++++++++++++++++
src/backend/optimizer/plan/planner.c | 6 +-
src/backend/optimizer/util/appendinfo.c | 12 ++-
src/backend/optimizer/util/relnode.c | 14 +--
src/include/optimizer/paths.h | 10 +-
src/test/regress/expected/partition_join.out | 144 +++++++++++++++++++++++++++
src/test/regress/sql/partition_join.sql | 64 ++++++++++++
9 files changed, 376 insertions(+), 21 deletions(-)
diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c
index db3a68a..69c3cb2 100644
--- a/src/backend/optimizer/path/allpaths.c
+++ b/src/backend/optimizer/path/allpaths.c
@@ -1275,7 +1275,7 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel,
}
/* Add paths to the append relation. */
- add_paths_to_append_rel(root, rel, live_childrels);
+ add_paths_to_append_rel(root, rel, live_childrels, NIL);
}
@@ -1292,7 +1292,8 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel,
*/
void
add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel,
- List *live_childrels)
+ List *live_childrels,
+ List *original_partitioned_rels)
{
List *subpaths = NIL;
bool subpaths_valid = true;
@@ -1304,7 +1305,7 @@ add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel,
List *all_child_pathkeys = NIL;
List *all_child_outers = NIL;
ListCell *l;
- List *partitioned_rels = NIL;
+ List *partitioned_rels = original_partitioned_rels;
double partial_rows = -1;
/* If appropriate, consider parallel append */
@@ -3717,7 +3718,7 @@ generate_partitionwise_join_paths(PlannerInfo *root, RelOptInfo *rel)
}
/* Build additional paths for this rel from child-join paths. */
- add_paths_to_append_rel(root, rel, live_children);
+ add_paths_to_append_rel(root, rel, live_children, NIL);
list_free(live_children);
}
diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c
index dc28b56..9444758 100644
--- a/src/backend/optimizer/path/joinpath.c
+++ b/src/backend/optimizer/path/joinpath.c
@@ -324,6 +324,15 @@ add_paths_to_joinrel(PlannerInfo *root,
if (set_join_pathlist_hook)
set_join_pathlist_hook(root, joinrel, outerrel, innerrel,
jointype, &extra);
+
+ /*
+ * 7. If outer relation is delivered from partition-tables, consider
+ * distributing inner relation to every partition-leaf prior to
+ * append these leafs.
+ */
+ try_asymmetric_partitionwise_join(root, joinrel,
+ outerrel, innerrel,
+ jointype, &extra);
}
/*
diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c
index 6a480ab..6c35d35 100644
--- a/src/backend/optimizer/path/joinrels.c
+++ b/src/backend/optimizer/path/joinrels.c
@@ -16,6 +16,7 @@
#include "miscadmin.h"
#include "optimizer/appendinfo.h"
+#include "optimizer/cost.h"
#include "optimizer/joininfo.h"
#include "optimizer/pathnode.h"
#include "optimizer/paths.h"
@@ -1532,6 +1533,134 @@ try_partitionwise_join(PlannerInfo *root, RelOptInfo *rel1, RelOptInfo *rel2,
}
}
+static List *
+extract_asymmetric_partitionwise_subjoin(PlannerInfo *root,
+ RelOptInfo *joinrel,
+ AppendPath *append_path,
+ RelOptInfo *inner_rel,
+ JoinType jointype,
+ JoinPathExtraData *extra)
+{
+ List *result = NIL;
+ ListCell *lc;
+
+ foreach (lc, append_path->subpaths)
+ {
+ Path *child_path = lfirst(lc);
+ RelOptInfo *child_rel = child_path->parent;
+ Relids child_join_relids;
+ RelOptInfo *child_join_rel;
+ SpecialJoinInfo *child_sjinfo;
+ List *child_restrictlist;
+ AppendRelInfo **appinfos;
+ int nappinfos;
+
+ child_join_relids = bms_union(child_rel->relids,
+ inner_rel->relids);
+ appinfos = find_appinfos_by_relids(root, child_join_relids,
+ &nappinfos);
+ child_sjinfo = build_child_join_sjinfo(root, extra->sjinfo,
+ child_rel->relids,
+ inner_rel->relids);
+ child_restrictlist = (List *)
+ adjust_appendrel_attrs(root, (Node *)extra->restrictlist,
+ nappinfos, appinfos);
+ pfree(appinfos);
+
+ child_join_rel = find_join_rel(root, child_join_relids);
+ if (!child_join_rel)
+ {
+ child_join_rel = build_child_join_rel(root,
+ child_rel,
+ inner_rel,
+ joinrel,
+ child_restrictlist,
+ child_sjinfo,
+ jointype);
+ if (!child_join_rel)
+ return NIL;
+ }
+ populate_joinrel_with_paths(root,
+ child_rel,
+ inner_rel,
+ child_join_rel,
+ child_sjinfo,
+ child_restrictlist);
+ /* Give up if asymmetric partition-wise join is not available */
+ if (child_join_rel->pathlist == NIL)
+ return NIL;
+ set_cheapest(child_join_rel);
+
+ result = lappend(result, child_join_rel);
+ }
+ return result;
+}
+
+void
+try_asymmetric_partitionwise_join(PlannerInfo *root,
+ RelOptInfo *joinrel,
+ RelOptInfo *outer_rel,
+ RelOptInfo *inner_rel,
+ JoinType jointype,
+ JoinPathExtraData *extra)
+{
+ ListCell *lc;
+
+ if (!enable_partitionwise_join)
+ return;
+ if (IS_OTHER_REL(outer_rel) || IS_OTHER_REL(inner_rel))
+ return;
+ if (jointype != JOIN_INNER &&
+ jointype != JOIN_LEFT)
+ return;
+
+ foreach (lc, outer_rel->pathlist)
+ {
+ AppendPath *append_path = lfirst(lc);
+
+ /*
+ * MEMO: We assume this pathlist keeps at least one AppendPath that
+ * represents partitioned table-scan, symmetric or asymmetric
+ * partition-wise join. It is not correct right now, however, a hook
+ * on add_path() to give additional decision for path removel allows
+ * to retain this kind of AppendPath, regardless of its cost.
+ */
+ if (IsA(append_path, AppendPath) &&
+ append_path->partitioned_rels != NIL)
+ {
+ List **join_rel_level_saved;
+ List *live_childrels = NIL;
+
+ join_rel_level_saved = root->join_rel_level;
+ PG_TRY();
+ {
+ /* temporary disables "dynamic programming" algorithm */
+ root->join_rel_level = NULL;
+
+ live_childrels =
+ extract_asymmetric_partitionwise_subjoin(root,
+ joinrel,
+ append_path,
+ inner_rel,
+ jointype,
+ extra);
+ }
+ PG_CATCH();
+ {
+ root->join_rel_level = join_rel_level_saved;
+ PG_RE_THROW();
+ }
+ PG_END_TRY();
+ root->join_rel_level = join_rel_level_saved;
+
+ if (live_childrels != NIL)
+ add_paths_to_append_rel(root, joinrel, live_childrels,
+ append_path->partitioned_rels);
+ break;
+ }
+ }
+}
+
/*
* Construct the SpecialJoinInfo for a child-join by translating
* SpecialJoinInfo for the join between parents. left_relids and right_relids
diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c
index 17c5f08..c7a1fc9 100644
--- a/src/backend/optimizer/plan/planner.c
+++ b/src/backend/optimizer/plan/planner.c
@@ -7181,7 +7181,7 @@ apply_scanjoin_target_to_paths(PlannerInfo *root,
}
/* Build new paths for this relation by appending child paths. */
- add_paths_to_append_rel(root, rel, live_children);
+ add_paths_to_append_rel(root, rel, live_children, NIL);
}
/*
@@ -7334,7 +7334,7 @@ create_partitionwise_grouping_paths(PlannerInfo *root,
Assert(partially_grouped_live_children != NIL);
add_paths_to_append_rel(root, partially_grouped_rel,
- partially_grouped_live_children);
+ partially_grouped_live_children, NIL);
/*
* We need call set_cheapest, since the finalization step will use the
@@ -7349,7 +7349,7 @@ create_partitionwise_grouping_paths(PlannerInfo *root,
{
Assert(grouped_live_children != NIL);
- add_paths_to_append_rel(root, grouped_rel, grouped_live_children);
+ add_paths_to_append_rel(root, grouped_rel, grouped_live_children, NIL);
}
}
diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c
index 16d3151..7496cb9 100644
--- a/src/backend/optimizer/util/appendinfo.c
+++ b/src/backend/optimizer/util/appendinfo.c
@@ -187,8 +187,9 @@ adjust_appendrel_attrs(PlannerInfo *root, Node *node, int nappinfos,
context.nappinfos = nappinfos;
context.appinfos = appinfos;
- /* If there's nothing to adjust, don't call this function. */
- Assert(nappinfos >= 1 && appinfos != NULL);
+ /* If there's nothing to adjust, just return a duplication */
+ if (nappinfos == 0)
+ return copyObject(node);
/*
* Must be prepared to start with a Query or a bare expression tree.
@@ -709,11 +710,11 @@ AppendRelInfo **
find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos)
{
AppendRelInfo **appinfos;
+ int nrooms = bms_num_members(relids);
int cnt = 0;
int i;
- *nappinfos = bms_num_members(relids);
- appinfos = (AppendRelInfo **) palloc(sizeof(AppendRelInfo *) * *nappinfos);
+ appinfos = (AppendRelInfo **) palloc(sizeof(AppendRelInfo *) * nrooms);
i = -1;
while ((i = bms_next_member(relids, i)) >= 0)
@@ -721,9 +722,10 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos)
AppendRelInfo *appinfo = root->append_rel_array[i];
if (!appinfo)
- elog(ERROR, "child rel %d not found in append_rel_array", i);
+ continue;
appinfos[cnt++] = appinfo;
}
+ *nappinfos = cnt;
return appinfos;
}
diff --git a/src/backend/optimizer/util/relnode.c b/src/backend/optimizer/util/relnode.c
index 8541538..c1a36b5 100644
--- a/src/backend/optimizer/util/relnode.c
+++ b/src/backend/optimizer/util/relnode.c
@@ -778,11 +778,8 @@ build_child_join_rel(PlannerInfo *root, RelOptInfo *outer_rel,
AppendRelInfo **appinfos;
int nappinfos;
- /* Only joins between "other" relations land here. */
- Assert(IS_OTHER_REL(outer_rel) && IS_OTHER_REL(inner_rel));
-
- /* The parent joinrel should have consider_partitionwise_join set. */
- Assert(parent_joinrel->consider_partitionwise_join);
+ /* Either of relations must be "other" relation at least. */
+ Assert(IS_OTHER_REL(outer_rel) || IS_OTHER_REL(inner_rel));
joinrel->reloptkind = RELOPT_OTHER_JOINREL;
joinrel->relids = bms_union(outer_rel->relids, inner_rel->relids);
@@ -837,8 +834,11 @@ build_child_join_rel(PlannerInfo *root, RelOptInfo *outer_rel,
joinrel->nullable_partexprs = NULL;
joinrel->partitioned_child_rels = NIL;
- joinrel->top_parent_relids = bms_union(outer_rel->top_parent_relids,
- inner_rel->top_parent_relids);
+ joinrel->top_parent_relids =
+ bms_union(IS_OTHER_REL(outer_rel) ?
+ outer_rel->top_parent_relids : outer_rel->relids,
+ IS_OTHER_REL(inner_rel) ?
+ inner_rel->top_parent_relids : inner_rel->relids);
/* Compute information relevant to foreign relations. */
set_foreign_rel_properties(joinrel, outer_rel, inner_rel);
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index 7345137..e70ebe7 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -109,7 +109,12 @@ extern void mark_dummy_rel(RelOptInfo *rel);
extern bool have_partkey_equi_join(RelOptInfo *joinrel,
RelOptInfo *rel1, RelOptInfo *rel2,
JoinType jointype, List *restrictlist);
-
+extern void try_asymmetric_partitionwise_join(PlannerInfo *root,
+ RelOptInfo *joinrel,
+ RelOptInfo *outer_rel,
+ RelOptInfo *inner_rel,
+ JoinType jointype,
+ JoinPathExtraData *extra);
/*
* equivclass.c
* routines for managing EquivalenceClasses
@@ -233,6 +238,7 @@ extern PathKey *make_canonical_pathkey(PlannerInfo *root,
EquivalenceClass *eclass, Oid opfamily,
int strategy, bool nulls_first);
extern void add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel,
- List *live_childrels);
+ List *live_childrels,
+ List *original_partitioned_rels);
#endif /* PATHS_H */
diff --git a/src/test/regress/expected/partition_join.out b/src/test/regress/expected/partition_join.out
index 1296edc..d838713 100644
--- a/src/test/regress/expected/partition_join.out
+++ b/src/test/regress/expected/partition_join.out
@@ -2003,3 +2003,147 @@ SELECT t1.a, t1.c, t2.b, t2.c FROM prt1 t1, prt2 t2 WHERE t1.a = t2.b AND t1.b =
Filter: (b = 0)
(16 rows)
+--
+-- For asymmetric partition-wise join
+--
+CREATE TABLE prt5 (hkey int, a int, b int) PARTITION BY HASH(hkey);
+CREATE TABLE prt5_p0 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 0);
+CREATE TABLE prt5_p1 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 1);
+CREATE TABLE prt5_p2 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 2);
+CREATE TABLE t5_1 (aid int, alabel text);
+CREATE TABLE t5_2 (bid int, blabel text);
+INSERT INTO prt5 (SELECT x, (1000.0 * random())::int,
+ (1000.0 * random())::int
+ FROM generate_series(1,1000000) x);
+INSERT INTO t5_1 (SELECT x, md5(x::text) FROM generate_series(-200, 800) x);
+INSERT INTO t5_2 (SELECT x, md5(x::text) FROM generate_series(-200, 800) x);
+VACUUM ANALYZE prt5;
+VACUUM ANALYZE t5_1;
+VACUUM ANALYZE t5_2;
+SET max_parallel_workers_per_gather = 0;
+EXPLAIN (COSTS OFF)
+SELECT *
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+-------------------------------------------------------
+ Append
+ -> Hash Join
+ Hash Cond: (prt5_p0.a = t5_1.aid)
+ -> Seq Scan on prt5_p0
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_p1.a = t5_1.aid)
+ -> Seq Scan on prt5_p1
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_p2.a = t5_1.aid)
+ -> Seq Scan on prt5_p2
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+(19 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT *
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+ QUERY PLAN
+------------------------------------------------------------
+ Append
+ -> Hash Join
+ Hash Cond: (prt5_p0.a = t5_1.aid)
+ -> Hash Join
+ Hash Cond: (prt5_p0.b = t5_2.bid)
+ -> Seq Scan on prt5_p0
+ -> Hash
+ -> Seq Scan on t5_2
+ Filter: (blabel ~~ '%cd%'::text)
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%ab%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_p1.a = t5_1.aid)
+ -> Hash Join
+ Hash Cond: (prt5_p1.b = t5_2.bid)
+ -> Seq Scan on prt5_p1
+ -> Hash
+ -> Seq Scan on t5_2
+ Filter: (blabel ~~ '%cd%'::text)
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%ab%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_p2.a = t5_1.aid)
+ -> Hash Join
+ Hash Cond: (prt5_p2.b = t5_2.bid)
+ -> Seq Scan on prt5_p2
+ -> Hash
+ -> Seq Scan on t5_2
+ Filter: (blabel ~~ '%cd%'::text)
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%ab%'::text)
+(34 rows)
+
+-- unable to extract non-partitioned right relation
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 RIGHT JOIN t5_1 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+-----------------------------------------------
+ Hash Right Join
+ Hash Cond: (prt5_p0.a = t5_1.aid)
+ Join Filter: (t5_1.alabel ~~ '%abc%'::text)
+ -> Append
+ -> Seq Scan on prt5_p0
+ -> Seq Scan on prt5_p1
+ -> Seq Scan on prt5_p2
+ -> Hash
+ -> Seq Scan on t5_1
+(9 rows)
+
+-- left side can be extracted, but no cost benefit
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 LEFT JOIN t5_1 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+-------------------------------------------------
+ Hash Left Join
+ Hash Cond: (prt5_p0.a = t5_1.aid)
+ -> Append
+ -> Seq Scan on prt5_p0
+ -> Seq Scan on prt5_p1
+ -> Seq Scan on prt5_p2
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+(9 rows)
+
+-- validation of the results with/without asymmetric partition-wise join
+SELECT * INTO pg_temp.result01a
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+SELECT * INTO pg_temp.result02a
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+SET enable_partitionwise_join = off;
+SELECT * INTO pg_temp.result01b
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+SELECT * INTO pg_temp.result02b
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+SELECT * FROM pg_temp.result01a EXCEPT SELECT * FROM pg_temp.result01b;
+ hkey | a | b | aid | alabel
+------+---+---+-----+--------
+(0 rows)
+
+SELECT * FROM pg_temp.result02a EXCEPT SELECT * FROM pg_temp.result02b;
+ hkey | a | b | aid | alabel | bid | blabel
+------+---+---+-----+--------+-----+--------
+(0 rows)
+
+RESET max_parallel_workers_per_gather;
diff --git a/src/test/regress/sql/partition_join.sql b/src/test/regress/sql/partition_join.sql
index db9a6b4..8ad54b4 100644
--- a/src/test/regress/sql/partition_join.sql
+++ b/src/test/regress/sql/partition_join.sql
@@ -435,3 +435,67 @@ ANALYZE prt2;
EXPLAIN (COSTS OFF)
SELECT t1.a, t1.c, t2.b, t2.c FROM prt1 t1, prt2 t2 WHERE t1.a = t2.b AND t1.b = 0 ORDER BY t1.a, t2.b;
+
+--
+-- For asymmetric partition-wise join
+--
+CREATE TABLE prt5 (hkey int, a int, b int) PARTITION BY HASH(hkey);
+CREATE TABLE prt5_p0 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 0);
+CREATE TABLE prt5_p1 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 1);
+CREATE TABLE prt5_p2 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 2);
+CREATE TABLE t5_1 (aid int, alabel text);
+CREATE TABLE t5_2 (bid int, blabel text);
+
+INSERT INTO prt5 (SELECT x, (1000.0 * random())::int,
+ (1000.0 * random())::int
+ FROM generate_series(1,1000000) x);
+INSERT INTO t5_1 (SELECT x, md5(x::text) FROM generate_series(-200, 800) x);
+INSERT INTO t5_2 (SELECT x, md5(x::text) FROM generate_series(-200, 800) x);
+
+VACUUM ANALYZE prt5;
+VACUUM ANALYZE t5_1;
+VACUUM ANALYZE t5_2;
+
+SET max_parallel_workers_per_gather = 0;
+EXPLAIN (COSTS OFF)
+SELECT *
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+
+EXPLAIN (COSTS OFF)
+SELECT *
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+
+-- unable to extract non-partitioned right relation
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 RIGHT JOIN t5_1 ON a = aid AND alabel like '%abc%';
+-- left side can be extracted, but no cost benefit
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 LEFT JOIN t5_1 ON a = aid AND alabel like '%abc%';
+
+-- validation of the results with/without asymmetric partition-wise join
+SELECT * INTO pg_temp.result01a
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+
+SELECT * INTO pg_temp.result02a
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+
+SET enable_partitionwise_join = off;
+
+SELECT * INTO pg_temp.result01b
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+
+SELECT * INTO pg_temp.result02b
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+
+SELECT * FROM pg_temp.result01a EXCEPT SELECT * FROM pg_temp.result01b;
+SELECT * FROM pg_temp.result02a EXCEPT SELECT * FROM pg_temp.result02b;
+
+RESET max_parallel_workers_per_gather;
+
+
^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: Asymmetric partition-wise JOIN
2019-08-12 06:03 Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-08-22 16:05 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
@ 2019-08-23 22:01 ` Thomas Munro <[email protected]>
2019-08-24 08:33 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
0 siblings, 1 reply; 28+ messages in thread
From: Thomas Munro @ 2019-08-23 22:01 UTC (permalink / raw)
To: Kohei KaiGai <[email protected]>; +Cc: pgsql-hackers
On Fri, Aug 23, 2019 at 4:05 AM Kohei KaiGai <[email protected]> wrote:
> We can consider the table join ptable X t1 above is equivalent to:
> (ptable_p0 + ptable_p1 + ptable_p2) X t1
> = (ptable_p0 X t1) + (ptable_p1 X t1) + (ptable_p2 X t1)
> It returns an equivalent result, however, rows are already reduced by HashJoin
> in the individual leaf of Append, so CPU-cycles consumed by Append node can
> be cheaper.
>
> On the other hands, it has a downside because t1 must be read 3 times and
> hash table also must be built 3 times. It increases the expected cost,
> so planner
> may not choose the asymmetric partition-wise join plan.
What if you include the partition constraint as a filter on t1? So you get:
ptable X t1 =
(ptable_p0 X (σ hash(dist)%4=0 (t1))) +
(ptable_p1 X (σ hash(dist)%4=1 (t1))) +
(ptable_p2 X (σ hash(dist)%4=2 (t1))) +
(ptable_p3 X (σ hash(dist)%4=3 (t1)))
Pros:
1. The hash tables will not contain unnecessary junk.
2. You'll get the right answer if t1 is on the outer side of an outer join.
3. If this runs underneath a Parallel Append and t1 is big enough
then workers will hopefully cooperate and do a synchronised scan of
t1.
4. The filter might enable a selective and efficient plan like an index scan.
Cons:
1. The filter might not enable a selective and efficient plan, and
therefore cause extra work.
(It's a little weird in this example because don't usually see hash
functions in WHERE clauses, but that could just as easily be dist
BETWEEN 1 AND 99 or any other partition constraint.)
> One idea I have is, sibling HashJoin shares a hash table that was built once
> by any of the sibling Hash plan. Right now, it is not implemented yet.
Yeah, I've thought a little bit about that in the context of Parallel
Repartition. I'm interested in combining intra-node partitioning
(where a single plan node repartitions data among workers on the fly)
with inter-node partitioning (like PWJ, where partitions are handled
by different parts of the plan, considered at planning time); you
finish up needing to have nodes in the plan that 'receive' tuples for
each partition, to match up with the PWJ plan structure. That's not
entirely unlike CTE references, and not entirely unlike your idea of
somehow sharing the same hash table. I ran into a number of problems
while thinking about that, which I should write about in another
thread.
--
Thomas Munro
https://enterprisedb.com
^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: Asymmetric partition-wise JOIN
2019-08-12 06:03 Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-08-22 16:05 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-08-23 22:01 ` Re: Asymmetric partition-wise JOIN Thomas Munro <[email protected]>
@ 2019-08-24 08:33 ` Kohei KaiGai <[email protected]>
2019-12-01 03:24 ` Re: Asymmetric partition-wise JOIN Michael Paquier <[email protected]>
2020-08-26 13:32 ` Re: Asymmetric partition-wise JOIN Amul Sul <[email protected]>
0 siblings, 2 replies; 28+ messages in thread
From: Kohei KaiGai @ 2019-08-24 08:33 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: pgsql-hackers
2019年8月24日(土) 7:02 Thomas Munro <[email protected]>:
>
> On Fri, Aug 23, 2019 at 4:05 AM Kohei KaiGai <[email protected]> wrote:
> > We can consider the table join ptable X t1 above is equivalent to:
> > (ptable_p0 + ptable_p1 + ptable_p2) X t1
> > = (ptable_p0 X t1) + (ptable_p1 X t1) + (ptable_p2 X t1)
> > It returns an equivalent result, however, rows are already reduced by HashJoin
> > in the individual leaf of Append, so CPU-cycles consumed by Append node can
> > be cheaper.
> >
> > On the other hands, it has a downside because t1 must be read 3 times and
> > hash table also must be built 3 times. It increases the expected cost,
> > so planner
> > may not choose the asymmetric partition-wise join plan.
>
> What if you include the partition constraint as a filter on t1? So you get:
>
> ptable X t1 =
> (ptable_p0 X (σ hash(dist)%4=0 (t1))) +
> (ptable_p1 X (σ hash(dist)%4=1 (t1))) +
> (ptable_p2 X (σ hash(dist)%4=2 (t1))) +
> (ptable_p3 X (σ hash(dist)%4=3 (t1)))
>
> Pros:
> 1. The hash tables will not contain unnecessary junk.
> 2. You'll get the right answer if t1 is on the outer side of an outer join.
> 3. If this runs underneath a Parallel Append and t1 is big enough
> then workers will hopefully cooperate and do a synchronised scan of
> t1.
> 4. The filter might enable a selective and efficient plan like an index scan.
>
> Cons:
> 1. The filter might not enable a selective and efficient plan, and
> therefore cause extra work.
>
> (It's a little weird in this example because don't usually see hash
> functions in WHERE clauses, but that could just as easily be dist
> BETWEEN 1 AND 99 or any other partition constraint.)
>
It requires the join-key must include the partition key and also must be
equality-join, doesn't it?
If ptable and t1 are joined using ptable.dist = t1.foo, we can distribute
t1 for each leaf table with "WHERE hash(foo)%4 = xxx" according to
the partition bounds, indeed.
In case when some of partition leafs are pruned, it is exactly beneficial
because relevant rows to be referenced by the pruned child relations
are waste of memory.
On the other hands, it eventually consumes almost equivalent amount
of memory to load the inner relations, if no leafs are pruned, and if we
could extend the Hash-node to share the hash-table with sibling join-nodess.
> > One idea I have is, sibling HashJoin shares a hash table that was built once
> > by any of the sibling Hash plan. Right now, it is not implemented yet.
>
> Yeah, I've thought a little bit about that in the context of Parallel
> Repartition. I'm interested in combining intra-node partitioning
> (where a single plan node repartitions data among workers on the fly)
> with inter-node partitioning (like PWJ, where partitions are handled
> by different parts of the plan, considered at planning time); you
> finish up needing to have nodes in the plan that 'receive' tuples for
> each partition, to match up with the PWJ plan structure. That's not
> entirely unlike CTE references, and not entirely unlike your idea of
> somehow sharing the same hash table. I ran into a number of problems
> while thinking about that, which I should write about in another
> thread.
>
Hmm. Do you intend the inner-path may have different behavior according
to the partition bounds definition where the outer-path to be joined?
Let me investigate its pros & cons.
The reasons why I think the idea of sharing the same hash table is reasonable
in this scenario are:
1. We can easily extend the idea for parallel optimization. A hash table on DSM
segment, once built, can be shared by all the siblings in all the
parallel workers.
2. We can save the memory consumption regardless of the join-keys and
partition-keys, even if these are not involved in the query.
On the other hands, below are the downside. Potentially, combined use of
your idea may help these cases:
3. Distributed inner-relation cannot be outer side of XXX OUTER JOIN.
4. Hash table contains rows to be referenced by only pruned partition leafs.
Best regards,
--
HeteroDB, Inc / The PG-Strom Project
KaiGai Kohei <[email protected]>
^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: Asymmetric partition-wise JOIN
2019-08-12 06:03 Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-08-22 16:05 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-08-23 22:01 ` Re: Asymmetric partition-wise JOIN Thomas Munro <[email protected]>
2019-08-24 08:33 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
@ 2019-12-01 03:24 ` Michael Paquier <[email protected]>
2019-12-27 07:34 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
1 sibling, 1 reply; 28+ messages in thread
From: Michael Paquier @ 2019-12-01 03:24 UTC (permalink / raw)
To: Kohei KaiGai <[email protected]>; +Cc: Thomas Munro <[email protected]>; pgsql-hackers
On Sat, Aug 24, 2019 at 05:33:01PM +0900, Kohei KaiGai wrote:
> On the other hands, it eventually consumes almost equivalent amount
> of memory to load the inner relations, if no leafs are pruned, and if we
> could extend the Hash-node to share the hash-table with sibling
> join-nodess.
The patch crashes when running the regression tests, per the report of
the automatic patch tester. Could you look at that? I have moved the
patch to nexf CF, waiting on author.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: Asymmetric partition-wise JOIN
2019-08-12 06:03 Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-08-22 16:05 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-08-23 22:01 ` Re: Asymmetric partition-wise JOIN Thomas Munro <[email protected]>
2019-08-24 08:33 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-12-01 03:24 ` Re: Asymmetric partition-wise JOIN Michael Paquier <[email protected]>
@ 2019-12-27 07:34 ` Kohei KaiGai <[email protected]>
2020-03-27 14:44 ` Re: Asymmetric partition-wise JOIN David Steele <[email protected]>
2020-07-01 09:10 ` Re: Asymmetric partition-wise JOIN Daniel Gustafsson <[email protected]>
2020-07-06 08:46 ` Re: Asymmetric partition-wise JOIN Andrey V. Lepikhov <[email protected]>
0 siblings, 3 replies; 28+ messages in thread
From: Kohei KaiGai @ 2019-12-27 07:34 UTC (permalink / raw)
To: Michael Paquier <[email protected]>; +Cc: Thomas Munro <[email protected]>; pgsql-hackers
Hello,
This crash was reproduced on our environment also.
It looks to me adjust_child_relids_multilevel() didn't expect a case
when supplied 'relids'
(partially) indicate normal and non-partitioned relation.
It tries to build a new 'parent_relids' that is a set of
appinfo->parent_relid related to the
supplied 'child_relids'. However, bits in child_relids that indicate
normal relations are
unintentionally dropped here. Then, adjust_child_relids_multilevel()
goes to an infinite
recursion until stack limitation.
The attached v2 fixed the problem, and regression test finished correctly.
Best regards,
2019年12月1日(日) 12:24 Michael Paquier <[email protected]>:
>
> On Sat, Aug 24, 2019 at 05:33:01PM +0900, Kohei KaiGai wrote:
> > On the other hands, it eventually consumes almost equivalent amount
> > of memory to load the inner relations, if no leafs are pruned, and if we
> > could extend the Hash-node to share the hash-table with sibling
> > join-nodess.
>
> The patch crashes when running the regression tests, per the report of
> the automatic patch tester. Could you look at that? I have moved the
> patch to nexf CF, waiting on author.
> --
> Michael
--
HeteroDB, Inc / The PG-Strom Project
KaiGai Kohei <[email protected]>
#12912 0x00000000006f9471 in adjust_child_relids_multilevel (
root=root@entry=0x2115240, relids=relids@entry=0x2106df8,
child_relids=child_relids@entry=0x0,
top_parent_relids=top_parent_relids@entry=0x20afd88) at appendinfo.c:602
#12913 0x00000000006f9471 in adjust_child_relids_multilevel (
root=root@entry=0x2115240, relids=relids@entry=0x2106df8,
child_relids=child_relids@entry=0x0,
top_parent_relids=top_parent_relids@entry=0x20afd88) at appendinfo.c:602
#12914 0x00000000006f9471 in adjust_child_relids_multilevel (
root=root@entry=0x2115240, relids=relids@entry=0x2106df8,
child_relids=child_relids@entry=0x0,
top_parent_relids=top_parent_relids@entry=0x20afd88) at appendinfo.c:602
#12915 0x00000000006f9471 in adjust_child_relids_multilevel (
root=root@entry=0x2115240, relids=relids@entry=0x2106df8,
child_relids=child_relids@entry=0x20b0bb8,
top_parent_relids=top_parent_relids@entry=0x20afd88) at appendinfo.c:602
#12916 0x00000000006f9471 in adjust_child_relids_multilevel (
root=root@entry=0x2115240, relids=relids@entry=0x2106df8,
child_relids=child_relids@entry=0x20af9d8,
top_parent_relids=top_parent_relids@entry=0x20afd88) at appendinfo.c:602
#12917 0x00000000006c9805 in add_child_join_rel_equivalences (
root=root@entry=0x2115240, nappinfos=1, appinfos=appinfos@entry=0x20afdb0,
parent_joinrel=parent_joinrel@entry=0x20ae420,
child_joinrel=child_joinrel@entry=0x20aef10) at equivclass.c:2458
#12918 0x000000000070bc07 in build_child_join_rel (root=root@entry=0x2115240,
outer_rel=outer_rel@entry=0x210a318, inner_rel=inner_rel@entry=0x21051a0,
parent_joinrel=parent_joinrel@entry=0x20ae420,
restrictlist=restrictlist@entry=0x20afcd8, sjinfo=sjinfo@entry=0x20afa00,
jointype=JOIN_INNER) at relnode.c:896
#12919 0x00000000006d2dcf in extract_asymmetric_partitionwise_subjoin (
append_path=0x20a4000, extra=0x7ffea461c710, extra=0x7ffea461c710,
jointype=JOIN_INNER, inner_rel=0x21051a0, joinrel=0x20ae420,
root=0x2115240) at joinrels.c:1573
#12920 try_asymmetric_partitionwise_join (root=root@entry=0x2115240,
joinrel=joinrel@entry=0x20ae420, outer_rel=outer_rel@entry=0x207cce0,
inner_rel=inner_rel@entry=0x21051a0, jointype=jointype@entry=JOIN_INNER,
extra=extra@entry=0x7ffea461c710) at joinrels.c:1641
#12921 0x00000000006cf296 in add_paths_to_joinrel (root=root@entry=0x2115240,
joinrel=joinrel@entry=0x20ae420, outerrel=outerrel@entry=0x207cce0,
innerrel=innerrel@entry=0x21051a0, jointype=jointype@entry=JOIN_INNER,
sjinfo=sjinfo@entry=0x7ffea461c8c0, restrictlist=0x20ae898)
at joinpath.c:333
#12922 0x00000000006d1a3a in populate_joinrel_with_paths (root=0x2115240,
rel1=0x207cce0, rel2=0x21051a0, joinrel=0x20ae420, sjinfo=0x7ffea461c8c0,
restrictlist=0x20ae898) at joinrels.c:804
#12923 0x00000000006d2505 in make_join_rel (root=root@entry=0x2115240,
rel1=rel1@entry=0x207cce0, rel2=rel2@entry=0x21051a0) at joinrels.c:757
#12924 0x00000000006d278d in make_rels_by_clause_joins (
other_rels=<optimized out>, other_rels_list=0x20ae390, old_rel=0x207cce0,
root=0x2115240) at joinrels.c:309
#12925 join_search_one_level (root=root@entry=0x2115240, level=level@entry=2)
at joinrels.c:120
#12926 0x00000000006bdd4b in standard_join_search (root=0x2115240,
levels_needed=3, initial_rels=<optimized out>) at allpaths.c:2879
#12927 0x00000000006be1c4 in make_one_rel (root=root@entry=0x2115240,
joinlist=joinlist@entry=0x2106b80) at allpaths.c:227
#12928 0x00000000006e1cdb in query_planner (root=root@entry=0x2115240,
qp_callback=qp_callback@entry=0x6e22b0 <standard_qp_callback>,
qp_extra=qp_extra@entry=0x7ffea461cb60) at planmain.c:269
#12929 0x00000000006e6962 in grouping_planner (root=<optimized out>,
inheritance_update=false, tuple_fraction=<optimized out>) at planner.c:2059
#12930 0x00000000006e9952 in subquery_planner (glob=glob@entry=0x2114e78,
parse=parse@entry=0x2085558, parent_root=parent_root@entry=0x0,
hasRecursion=hasRecursion@entry=false,
tuple_fraction=tuple_fraction@entry=0) at planner.c:1016
#12931 0x00000000006eaaaf in standard_planner (parse=0x2085558,
cursorOptions=256, boundParams=<optimized out>) at planner.c:406
#12932 0x00000000007aa059 in pg_plan_query (
querytree=querytree@entry=0x2085558,
cursorOptions=cursorOptions@entry=256, boundParams=boundParams@entry=0x0)
at postgres.c:874
#12933 0x00000000005d7433 in ExplainOneQuery (query=0x2085558,
cursorOptions=256, into=0x0, es=0x207cc10,
queryString=0x1fa95b0 "EXPLAIN (COSTS OFF)\nSELECT t1.a, ss.t2a, ss.t2c FROM prt1 t1 LEFT JOIN LATERAL\n\t\t\t (SELECT t2.a AS t2a, t3.a AS t3a, t2.b t2b, t2.c t2c, least(t1.a,t2.a,t3.b) FROM prt1 t2 JOIN prt2 t3 ON (t2.a = t3."...,
params=0x0, queryEnv=0x0) at explain.c:367
#12934 0x00000000005d7a5b in ExplainQuery (pstate=pstate@entry=0x207cab0,
stmt=stmt@entry=0x206d918,
queryString=queryString@entry=0x1fa95b0 "EXPLAIN (COSTS OFF)\nSELECT t1.a, ss.t2a, ss.t2c FROM prt1 t1 LEFT JOIN LATERAL\n\t\t\t (SELECT t2.a AS t2a, t3.a AS t3a, t2.b t2b, t2.c t2c, least(t1.a,t2.a,t3.b) FROM prt1 t2 JOIN prt2 t3 ON (t2.a = t3."..., params=params@entry=0x0, queryEnv=queryEnv@entry=0x0,
dest=dest@entry=0x207ca18) at ../../../src/include/nodes/nodes.h:590
#12935 0x00000000007afc6d in standard_ProcessUtility (pstmt=0x206d9e0,
queryString=0x1fa95b0 "EXPLAIN (COSTS OFF)\nSELECT t1.a, ss.t2a, ss.t2c FROM prt1 t1 LEFT JOIN LATERAL\n\t\t\t (SELECT t2.a AS t2a, t3.a AS t3a, t2.b t2b, t2.c t2c, least(t1.a,t2.a,t3.b) FROM prt1 t2 JOIN prt2 t3 ON (t2.a = t3."...,
context=PROCESS_UTILITY_TOPLEVEL, params=0x0, queryEnv=0x0,
dest=0x207ca18, completionTag=0x7ffea461cfc0 "") at utility.c:670
#12936 0x00000000007ad044 in PortalRunUtility (portal=0x20108f0,
pstmt=0x206d9e0, isTopLevel=<optimized out>,
setHoldSnapshot=<optimized out>, dest=0x207ca18,
completionTag=0x7ffea461cfc0 "") at pquery.c:1175
#12937 0x00000000007adf76 in FillPortalStore (portal=0x20108f0,
isTopLevel=<optimized out>) at ../../../src/include/nodes/nodes.h:590
#12938 0x00000000007aeb05 in PortalRun (portal=portal@entry=0x20108f0,
count=count@entry=9223372036854775807, isTopLevel=isTopLevel@entry=true,
run_once=run_once@entry=true, dest=dest@entry=0x20e9948,
altdest=altdest@entry=0x20e9948, completionTag=0x7ffea461d1e0 "")
at pquery.c:765
#12939 0x00000000007aa551 in exec_simple_query (
query_string=0x1fa95b0 "EXPLAIN (COSTS OFF)\nSELECT t1.a, ss.t2a, ss.t2c FROM prt1 t1 LEFT JOIN LATERAL\n\t\t\t (SELECT t2.a AS t2a, t3.a AS t3a, t2.b t2b, t2.c t2c, least(t1.a,t2.a,t3.b) FROM prt1 t2 JOIN prt2 t3 ON (t2.a = t3."...)
at postgres.c:1227
#12940 0x00000000007abfed in PostgresMain (argc=<optimized out>,
argv=argv@entry=0x1fd4b38, dbname=<optimized out>,
username=<optimized out>) at postgres.c:4291
#12941 0x000000000072acef in BackendRun (port=0x1fcb2b0, port=0x1fcb2b0)
at postmaster.c:4498
#12942 BackendStartup (port=0x1fcb2b0) at postmaster.c:4189
#12943 ServerLoop () at postmaster.c:1727
#12944 0x000000000072bbff in PostmasterMain (argc=argc@entry=8,
argv=argv@entry=0x1fa5160) at postmaster.c:1400
#12945 0x000000000047d2be in main (argc=8, argv=0x1fa5160) at main.c:210
Attachments:
[text/plain] gdb_back_trace.txt (7.1K, ../../CAOP8fzaDAP_6dR0VbCQrc5smcdzvwx5vEafs6bNhEesP0VP2KA@mail.gmail.com/2-gdb_back_trace.txt)
download | inline:
#12912 0x00000000006f9471 in adjust_child_relids_multilevel (
root=root@entry=0x2115240, relids=relids@entry=0x2106df8,
child_relids=child_relids@entry=0x0,
top_parent_relids=top_parent_relids@entry=0x20afd88) at appendinfo.c:602
#12913 0x00000000006f9471 in adjust_child_relids_multilevel (
root=root@entry=0x2115240, relids=relids@entry=0x2106df8,
child_relids=child_relids@entry=0x0,
top_parent_relids=top_parent_relids@entry=0x20afd88) at appendinfo.c:602
#12914 0x00000000006f9471 in adjust_child_relids_multilevel (
root=root@entry=0x2115240, relids=relids@entry=0x2106df8,
child_relids=child_relids@entry=0x0,
top_parent_relids=top_parent_relids@entry=0x20afd88) at appendinfo.c:602
#12915 0x00000000006f9471 in adjust_child_relids_multilevel (
root=root@entry=0x2115240, relids=relids@entry=0x2106df8,
child_relids=child_relids@entry=0x20b0bb8,
top_parent_relids=top_parent_relids@entry=0x20afd88) at appendinfo.c:602
#12916 0x00000000006f9471 in adjust_child_relids_multilevel (
root=root@entry=0x2115240, relids=relids@entry=0x2106df8,
child_relids=child_relids@entry=0x20af9d8,
top_parent_relids=top_parent_relids@entry=0x20afd88) at appendinfo.c:602
#12917 0x00000000006c9805 in add_child_join_rel_equivalences (
root=root@entry=0x2115240, nappinfos=1, appinfos=appinfos@entry=0x20afdb0,
parent_joinrel=parent_joinrel@entry=0x20ae420,
child_joinrel=child_joinrel@entry=0x20aef10) at equivclass.c:2458
#12918 0x000000000070bc07 in build_child_join_rel (root=root@entry=0x2115240,
outer_rel=outer_rel@entry=0x210a318, inner_rel=inner_rel@entry=0x21051a0,
parent_joinrel=parent_joinrel@entry=0x20ae420,
restrictlist=restrictlist@entry=0x20afcd8, sjinfo=sjinfo@entry=0x20afa00,
jointype=JOIN_INNER) at relnode.c:896
#12919 0x00000000006d2dcf in extract_asymmetric_partitionwise_subjoin (
append_path=0x20a4000, extra=0x7ffea461c710, extra=0x7ffea461c710,
jointype=JOIN_INNER, inner_rel=0x21051a0, joinrel=0x20ae420,
root=0x2115240) at joinrels.c:1573
#12920 try_asymmetric_partitionwise_join (root=root@entry=0x2115240,
joinrel=joinrel@entry=0x20ae420, outer_rel=outer_rel@entry=0x207cce0,
inner_rel=inner_rel@entry=0x21051a0, jointype=jointype@entry=JOIN_INNER,
extra=extra@entry=0x7ffea461c710) at joinrels.c:1641
#12921 0x00000000006cf296 in add_paths_to_joinrel (root=root@entry=0x2115240,
joinrel=joinrel@entry=0x20ae420, outerrel=outerrel@entry=0x207cce0,
innerrel=innerrel@entry=0x21051a0, jointype=jointype@entry=JOIN_INNER,
sjinfo=sjinfo@entry=0x7ffea461c8c0, restrictlist=0x20ae898)
at joinpath.c:333
#12922 0x00000000006d1a3a in populate_joinrel_with_paths (root=0x2115240,
rel1=0x207cce0, rel2=0x21051a0, joinrel=0x20ae420, sjinfo=0x7ffea461c8c0,
restrictlist=0x20ae898) at joinrels.c:804
#12923 0x00000000006d2505 in make_join_rel (root=root@entry=0x2115240,
rel1=rel1@entry=0x207cce0, rel2=rel2@entry=0x21051a0) at joinrels.c:757
#12924 0x00000000006d278d in make_rels_by_clause_joins (
other_rels=<optimized out>, other_rels_list=0x20ae390, old_rel=0x207cce0,
root=0x2115240) at joinrels.c:309
#12925 join_search_one_level (root=root@entry=0x2115240, level=level@entry=2)
at joinrels.c:120
#12926 0x00000000006bdd4b in standard_join_search (root=0x2115240,
levels_needed=3, initial_rels=<optimized out>) at allpaths.c:2879
#12927 0x00000000006be1c4 in make_one_rel (root=root@entry=0x2115240,
joinlist=joinlist@entry=0x2106b80) at allpaths.c:227
#12928 0x00000000006e1cdb in query_planner (root=root@entry=0x2115240,
qp_callback=qp_callback@entry=0x6e22b0 <standard_qp_callback>,
qp_extra=qp_extra@entry=0x7ffea461cb60) at planmain.c:269
#12929 0x00000000006e6962 in grouping_planner (root=<optimized out>,
inheritance_update=false, tuple_fraction=<optimized out>) at planner.c:2059
#12930 0x00000000006e9952 in subquery_planner (glob=glob@entry=0x2114e78,
parse=parse@entry=0x2085558, parent_root=parent_root@entry=0x0,
hasRecursion=hasRecursion@entry=false,
tuple_fraction=tuple_fraction@entry=0) at planner.c:1016
#12931 0x00000000006eaaaf in standard_planner (parse=0x2085558,
cursorOptions=256, boundParams=<optimized out>) at planner.c:406
#12932 0x00000000007aa059 in pg_plan_query (
querytree=querytree@entry=0x2085558,
cursorOptions=cursorOptions@entry=256, boundParams=boundParams@entry=0x0)
at postgres.c:874
#12933 0x00000000005d7433 in ExplainOneQuery (query=0x2085558,
cursorOptions=256, into=0x0, es=0x207cc10,
queryString=0x1fa95b0 "EXPLAIN (COSTS OFF)\nSELECT t1.a, ss.t2a, ss.t2c FROM prt1 t1 LEFT JOIN LATERAL\n\t\t\t (SELECT t2.a AS t2a, t3.a AS t3a, t2.b t2b, t2.c t2c, least(t1.a,t2.a,t3.b) FROM prt1 t2 JOIN prt2 t3 ON (t2.a = t3."...,
params=0x0, queryEnv=0x0) at explain.c:367
#12934 0x00000000005d7a5b in ExplainQuery (pstate=pstate@entry=0x207cab0,
stmt=stmt@entry=0x206d918,
queryString=queryString@entry=0x1fa95b0 "EXPLAIN (COSTS OFF)\nSELECT t1.a, ss.t2a, ss.t2c FROM prt1 t1 LEFT JOIN LATERAL\n\t\t\t (SELECT t2.a AS t2a, t3.a AS t3a, t2.b t2b, t2.c t2c, least(t1.a,t2.a,t3.b) FROM prt1 t2 JOIN prt2 t3 ON (t2.a = t3."..., params=params@entry=0x0, queryEnv=queryEnv@entry=0x0,
dest=dest@entry=0x207ca18) at ../../../src/include/nodes/nodes.h:590
#12935 0x00000000007afc6d in standard_ProcessUtility (pstmt=0x206d9e0,
queryString=0x1fa95b0 "EXPLAIN (COSTS OFF)\nSELECT t1.a, ss.t2a, ss.t2c FROM prt1 t1 LEFT JOIN LATERAL\n\t\t\t (SELECT t2.a AS t2a, t3.a AS t3a, t2.b t2b, t2.c t2c, least(t1.a,t2.a,t3.b) FROM prt1 t2 JOIN prt2 t3 ON (t2.a = t3."...,
context=PROCESS_UTILITY_TOPLEVEL, params=0x0, queryEnv=0x0,
dest=0x207ca18, completionTag=0x7ffea461cfc0 "") at utility.c:670
#12936 0x00000000007ad044 in PortalRunUtility (portal=0x20108f0,
pstmt=0x206d9e0, isTopLevel=<optimized out>,
setHoldSnapshot=<optimized out>, dest=0x207ca18,
completionTag=0x7ffea461cfc0 "") at pquery.c:1175
#12937 0x00000000007adf76 in FillPortalStore (portal=0x20108f0,
isTopLevel=<optimized out>) at ../../../src/include/nodes/nodes.h:590
#12938 0x00000000007aeb05 in PortalRun (portal=portal@entry=0x20108f0,
count=count@entry=9223372036854775807, isTopLevel=isTopLevel@entry=true,
run_once=run_once@entry=true, dest=dest@entry=0x20e9948,
altdest=altdest@entry=0x20e9948, completionTag=0x7ffea461d1e0 "")
at pquery.c:765
#12939 0x00000000007aa551 in exec_simple_query (
query_string=0x1fa95b0 "EXPLAIN (COSTS OFF)\nSELECT t1.a, ss.t2a, ss.t2c FROM prt1 t1 LEFT JOIN LATERAL\n\t\t\t (SELECT t2.a AS t2a, t3.a AS t3a, t2.b t2b, t2.c t2c, least(t1.a,t2.a,t3.b) FROM prt1 t2 JOIN prt2 t3 ON (t2.a = t3."...)
at postgres.c:1227
#12940 0x00000000007abfed in PostgresMain (argc=<optimized out>,
argv=argv@entry=0x1fd4b38, dbname=<optimized out>,
username=<optimized out>) at postgres.c:4291
#12941 0x000000000072acef in BackendRun (port=0x1fcb2b0, port=0x1fcb2b0)
at postmaster.c:4498
#12942 BackendStartup (port=0x1fcb2b0) at postmaster.c:4189
#12943 ServerLoop () at postmaster.c:1727
#12944 0x000000000072bbff in PostmasterMain (argc=argc@entry=8,
argv=argv@entry=0x1fa5160) at postmaster.c:1400
#12945 0x000000000047d2be in main (argc=8, argv=0x1fa5160) at main.c:210
[application/octet-stream] pgsql13-asymmetric-partitionwise-join.v2.patch (21.1K, ../../CAOP8fzaDAP_6dR0VbCQrc5smcdzvwx5vEafs6bNhEesP0VP2KA@mail.gmail.com/3-pgsql13-asymmetric-partitionwise-join.v2.patch)
download | inline diff:
src/backend/optimizer/path/allpaths.c | 9 +-
src/backend/optimizer/path/joinpath.c | 9 ++
src/backend/optimizer/path/joinrels.c | 129 ++++++++++++++++++++++++
src/backend/optimizer/plan/planner.c | 6 +-
src/backend/optimizer/util/appendinfo.c | 18 +++-
src/backend/optimizer/util/relnode.c | 14 +--
src/include/optimizer/paths.h | 10 +-
src/test/regress/expected/partition_join.out | 144 +++++++++++++++++++++++++++
src/test/regress/sql/partition_join.sql | 64 ++++++++++++
9 files changed, 382 insertions(+), 21 deletions(-)
diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c
index db3a68a51d..69c3cb2e8d 100644
--- a/src/backend/optimizer/path/allpaths.c
+++ b/src/backend/optimizer/path/allpaths.c
@@ -1275,7 +1275,7 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel,
}
/* Add paths to the append relation. */
- add_paths_to_append_rel(root, rel, live_childrels);
+ add_paths_to_append_rel(root, rel, live_childrels, NIL);
}
@@ -1292,7 +1292,8 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel,
*/
void
add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel,
- List *live_childrels)
+ List *live_childrels,
+ List *original_partitioned_rels)
{
List *subpaths = NIL;
bool subpaths_valid = true;
@@ -1304,7 +1305,7 @@ add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel,
List *all_child_pathkeys = NIL;
List *all_child_outers = NIL;
ListCell *l;
- List *partitioned_rels = NIL;
+ List *partitioned_rels = original_partitioned_rels;
double partial_rows = -1;
/* If appropriate, consider parallel append */
@@ -3717,7 +3718,7 @@ generate_partitionwise_join_paths(PlannerInfo *root, RelOptInfo *rel)
}
/* Build additional paths for this rel from child-join paths. */
- add_paths_to_append_rel(root, rel, live_children);
+ add_paths_to_append_rel(root, rel, live_children, NIL);
list_free(live_children);
}
diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c
index dc28b56e74..9444758a08 100644
--- a/src/backend/optimizer/path/joinpath.c
+++ b/src/backend/optimizer/path/joinpath.c
@@ -324,6 +324,15 @@ add_paths_to_joinrel(PlannerInfo *root,
if (set_join_pathlist_hook)
set_join_pathlist_hook(root, joinrel, outerrel, innerrel,
jointype, &extra);
+
+ /*
+ * 7. If outer relation is delivered from partition-tables, consider
+ * distributing inner relation to every partition-leaf prior to
+ * append these leafs.
+ */
+ try_asymmetric_partitionwise_join(root, joinrel,
+ outerrel, innerrel,
+ jointype, &extra);
}
/*
diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c
index 6a480ab764..6c35d35651 100644
--- a/src/backend/optimizer/path/joinrels.c
+++ b/src/backend/optimizer/path/joinrels.c
@@ -16,6 +16,7 @@
#include "miscadmin.h"
#include "optimizer/appendinfo.h"
+#include "optimizer/cost.h"
#include "optimizer/joininfo.h"
#include "optimizer/pathnode.h"
#include "optimizer/paths.h"
@@ -1532,6 +1533,134 @@ try_partitionwise_join(PlannerInfo *root, RelOptInfo *rel1, RelOptInfo *rel2,
}
}
+static List *
+extract_asymmetric_partitionwise_subjoin(PlannerInfo *root,
+ RelOptInfo *joinrel,
+ AppendPath *append_path,
+ RelOptInfo *inner_rel,
+ JoinType jointype,
+ JoinPathExtraData *extra)
+{
+ List *result = NIL;
+ ListCell *lc;
+
+ foreach (lc, append_path->subpaths)
+ {
+ Path *child_path = lfirst(lc);
+ RelOptInfo *child_rel = child_path->parent;
+ Relids child_join_relids;
+ RelOptInfo *child_join_rel;
+ SpecialJoinInfo *child_sjinfo;
+ List *child_restrictlist;
+ AppendRelInfo **appinfos;
+ int nappinfos;
+
+ child_join_relids = bms_union(child_rel->relids,
+ inner_rel->relids);
+ appinfos = find_appinfos_by_relids(root, child_join_relids,
+ &nappinfos);
+ child_sjinfo = build_child_join_sjinfo(root, extra->sjinfo,
+ child_rel->relids,
+ inner_rel->relids);
+ child_restrictlist = (List *)
+ adjust_appendrel_attrs(root, (Node *)extra->restrictlist,
+ nappinfos, appinfos);
+ pfree(appinfos);
+
+ child_join_rel = find_join_rel(root, child_join_relids);
+ if (!child_join_rel)
+ {
+ child_join_rel = build_child_join_rel(root,
+ child_rel,
+ inner_rel,
+ joinrel,
+ child_restrictlist,
+ child_sjinfo,
+ jointype);
+ if (!child_join_rel)
+ return NIL;
+ }
+ populate_joinrel_with_paths(root,
+ child_rel,
+ inner_rel,
+ child_join_rel,
+ child_sjinfo,
+ child_restrictlist);
+ /* Give up if asymmetric partition-wise join is not available */
+ if (child_join_rel->pathlist == NIL)
+ return NIL;
+ set_cheapest(child_join_rel);
+
+ result = lappend(result, child_join_rel);
+ }
+ return result;
+}
+
+void
+try_asymmetric_partitionwise_join(PlannerInfo *root,
+ RelOptInfo *joinrel,
+ RelOptInfo *outer_rel,
+ RelOptInfo *inner_rel,
+ JoinType jointype,
+ JoinPathExtraData *extra)
+{
+ ListCell *lc;
+
+ if (!enable_partitionwise_join)
+ return;
+ if (IS_OTHER_REL(outer_rel) || IS_OTHER_REL(inner_rel))
+ return;
+ if (jointype != JOIN_INNER &&
+ jointype != JOIN_LEFT)
+ return;
+
+ foreach (lc, outer_rel->pathlist)
+ {
+ AppendPath *append_path = lfirst(lc);
+
+ /*
+ * MEMO: We assume this pathlist keeps at least one AppendPath that
+ * represents partitioned table-scan, symmetric or asymmetric
+ * partition-wise join. It is not correct right now, however, a hook
+ * on add_path() to give additional decision for path removel allows
+ * to retain this kind of AppendPath, regardless of its cost.
+ */
+ if (IsA(append_path, AppendPath) &&
+ append_path->partitioned_rels != NIL)
+ {
+ List **join_rel_level_saved;
+ List *live_childrels = NIL;
+
+ join_rel_level_saved = root->join_rel_level;
+ PG_TRY();
+ {
+ /* temporary disables "dynamic programming" algorithm */
+ root->join_rel_level = NULL;
+
+ live_childrels =
+ extract_asymmetric_partitionwise_subjoin(root,
+ joinrel,
+ append_path,
+ inner_rel,
+ jointype,
+ extra);
+ }
+ PG_CATCH();
+ {
+ root->join_rel_level = join_rel_level_saved;
+ PG_RE_THROW();
+ }
+ PG_END_TRY();
+ root->join_rel_level = join_rel_level_saved;
+
+ if (live_childrels != NIL)
+ add_paths_to_append_rel(root, joinrel, live_childrels,
+ append_path->partitioned_rels);
+ break;
+ }
+ }
+}
+
/*
* Construct the SpecialJoinInfo for a child-join by translating
* SpecialJoinInfo for the join between parents. left_relids and right_relids
diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c
index d63ebb7287..22424e787e 100644
--- a/src/backend/optimizer/plan/planner.c
+++ b/src/backend/optimizer/plan/planner.c
@@ -7186,7 +7186,7 @@ apply_scanjoin_target_to_paths(PlannerInfo *root,
}
/* Build new paths for this relation by appending child paths. */
- add_paths_to_append_rel(root, rel, live_children);
+ add_paths_to_append_rel(root, rel, live_children, NIL);
}
/*
@@ -7339,7 +7339,7 @@ create_partitionwise_grouping_paths(PlannerInfo *root,
Assert(partially_grouped_live_children != NIL);
add_paths_to_append_rel(root, partially_grouped_rel,
- partially_grouped_live_children);
+ partially_grouped_live_children, NIL);
/*
* We need call set_cheapest, since the finalization step will use the
@@ -7354,7 +7354,7 @@ create_partitionwise_grouping_paths(PlannerInfo *root,
{
Assert(grouped_live_children != NIL);
- add_paths_to_append_rel(root, grouped_rel, grouped_live_children);
+ add_paths_to_append_rel(root, grouped_rel, grouped_live_children, NIL);
}
}
diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c
index 1890f256de..67b7470dbb 100644
--- a/src/backend/optimizer/util/appendinfo.c
+++ b/src/backend/optimizer/util/appendinfo.c
@@ -201,8 +201,9 @@ adjust_appendrel_attrs(PlannerInfo *root, Node *node, int nappinfos,
context.nappinfos = nappinfos;
context.appinfos = appinfos;
- /* If there's nothing to adjust, don't call this function. */
- Assert(nappinfos >= 1 && appinfos != NULL);
+ /* If there's nothing to adjust, just return a duplication */
+ if (nappinfos == 0)
+ return copyObject(node);
/*
* Must be prepared to start with a Query or a bare expression tree.
@@ -574,6 +575,7 @@ adjust_child_relids_multilevel(PlannerInfo *root, Relids relids,
AppendRelInfo **appinfos;
int nappinfos;
Relids parent_relids = NULL;
+ Relids normal_relids = NULL;
Relids result;
Relids tmp_result = NULL;
int cnt;
@@ -588,12 +590,17 @@ adjust_child_relids_multilevel(PlannerInfo *root, Relids relids,
appinfos = find_appinfos_by_relids(root, child_relids, &nappinfos);
/* Construct relids set for the immediate parent of the given child. */
+ normal_relids = bms_copy(child_relids);
for (cnt = 0; cnt < nappinfos; cnt++)
{
AppendRelInfo *appinfo = appinfos[cnt];
parent_relids = bms_add_member(parent_relids, appinfo->parent_relid);
+ normal_relids = bms_del_member(normal_relids, appinfo->child_relid);
}
+ parent_relids = bms_union(parent_relids, normal_relids);
+ if (normal_relids)
+ bms_free(normal_relids);
/* Recurse if immediate parent is not the top parent. */
if (!bms_equal(parent_relids, top_parent_relids))
@@ -723,11 +730,11 @@ AppendRelInfo **
find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos)
{
AppendRelInfo **appinfos;
+ int nrooms = bms_num_members(relids);
int cnt = 0;
int i;
- *nappinfos = bms_num_members(relids);
- appinfos = (AppendRelInfo **) palloc(sizeof(AppendRelInfo *) * *nappinfos);
+ appinfos = (AppendRelInfo **) palloc(sizeof(AppendRelInfo *) * nrooms);
i = -1;
while ((i = bms_next_member(relids, i)) >= 0)
@@ -735,9 +742,10 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos)
AppendRelInfo *appinfo = root->append_rel_array[i];
if (!appinfo)
- elog(ERROR, "child rel %d not found in append_rel_array", i);
+ continue;
appinfos[cnt++] = appinfo;
}
+ *nappinfos = cnt;
return appinfos;
}
diff --git a/src/backend/optimizer/util/relnode.c b/src/backend/optimizer/util/relnode.c
index c9eb447d07..2613e613c0 100644
--- a/src/backend/optimizer/util/relnode.c
+++ b/src/backend/optimizer/util/relnode.c
@@ -778,11 +778,8 @@ build_child_join_rel(PlannerInfo *root, RelOptInfo *outer_rel,
AppendRelInfo **appinfos;
int nappinfos;
- /* Only joins between "other" relations land here. */
- Assert(IS_OTHER_REL(outer_rel) && IS_OTHER_REL(inner_rel));
-
- /* The parent joinrel should have consider_partitionwise_join set. */
- Assert(parent_joinrel->consider_partitionwise_join);
+ /* Either of relations must be "other" relation at least. */
+ Assert(IS_OTHER_REL(outer_rel) || IS_OTHER_REL(inner_rel));
joinrel->reloptkind = RELOPT_OTHER_JOINREL;
joinrel->relids = bms_union(outer_rel->relids, inner_rel->relids);
@@ -837,8 +834,11 @@ build_child_join_rel(PlannerInfo *root, RelOptInfo *outer_rel,
joinrel->nullable_partexprs = NULL;
joinrel->partitioned_child_rels = NIL;
- joinrel->top_parent_relids = bms_union(outer_rel->top_parent_relids,
- inner_rel->top_parent_relids);
+ joinrel->top_parent_relids =
+ bms_union(IS_OTHER_REL(outer_rel) ?
+ outer_rel->top_parent_relids : outer_rel->relids,
+ IS_OTHER_REL(inner_rel) ?
+ inner_rel->top_parent_relids : inner_rel->relids);
/* Compute information relevant to foreign relations. */
set_foreign_rel_properties(joinrel, outer_rel, inner_rel);
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index c6c34630c2..07302a2b2d 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -109,7 +109,12 @@ extern void mark_dummy_rel(RelOptInfo *rel);
extern bool have_partkey_equi_join(RelOptInfo *joinrel,
RelOptInfo *rel1, RelOptInfo *rel2,
JoinType jointype, List *restrictlist);
-
+extern void try_asymmetric_partitionwise_join(PlannerInfo *root,
+ RelOptInfo *joinrel,
+ RelOptInfo *outer_rel,
+ RelOptInfo *inner_rel,
+ JoinType jointype,
+ JoinPathExtraData *extra);
/*
* equivclass.c
* routines for managing EquivalenceClasses
@@ -238,6 +243,7 @@ extern PathKey *make_canonical_pathkey(PlannerInfo *root,
EquivalenceClass *eclass, Oid opfamily,
int strategy, bool nulls_first);
extern void add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel,
- List *live_childrels);
+ List *live_childrels,
+ List *original_partitioned_rels);
#endif /* PATHS_H */
diff --git a/src/test/regress/expected/partition_join.out b/src/test/regress/expected/partition_join.out
index b3fbe47bde..da04359c91 100644
--- a/src/test/regress/expected/partition_join.out
+++ b/src/test/regress/expected/partition_join.out
@@ -2082,3 +2082,147 @@ SELECT t1.a, t1.c, t2.b, t2.c FROM prt1 t1, prt2 t2 WHERE t1.a = t2.b AND t1.b =
Filter: (b = 0)
(16 rows)
+--
+-- For asymmetric partition-wise join
+--
+CREATE TABLE prt5 (hkey int, a int, b int) PARTITION BY HASH(hkey);
+CREATE TABLE prt5_p0 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 0);
+CREATE TABLE prt5_p1 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 1);
+CREATE TABLE prt5_p2 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 2);
+CREATE TABLE t5_1 (aid int, alabel text);
+CREATE TABLE t5_2 (bid int, blabel text);
+INSERT INTO prt5 (SELECT x, (1000.0 * random())::int,
+ (1000.0 * random())::int
+ FROM generate_series(1,1000000) x);
+INSERT INTO t5_1 (SELECT x, md5(x::text) FROM generate_series(-200, 800) x);
+INSERT INTO t5_2 (SELECT x, md5(x::text) FROM generate_series(-200, 800) x);
+VACUUM ANALYZE prt5;
+VACUUM ANALYZE t5_1;
+VACUUM ANALYZE t5_2;
+SET max_parallel_workers_per_gather = 0;
+EXPLAIN (COSTS OFF)
+SELECT *
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+-------------------------------------------------------
+ Append
+ -> Hash Join
+ Hash Cond: (prt5_1.a = t5_1.aid)
+ -> Seq Scan on prt5_p0 prt5_1
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_2.a = t5_1.aid)
+ -> Seq Scan on prt5_p1 prt5_2
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_3.a = t5_1.aid)
+ -> Seq Scan on prt5_p2 prt5_3
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+(19 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT *
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+ QUERY PLAN
+------------------------------------------------------------
+ Append
+ -> Hash Join
+ Hash Cond: (prt5_1.a = t5_1.aid)
+ -> Hash Join
+ Hash Cond: (prt5_1.b = t5_2.bid)
+ -> Seq Scan on prt5_p0 prt5_1
+ -> Hash
+ -> Seq Scan on t5_2
+ Filter: (blabel ~~ '%cd%'::text)
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%ab%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_2.a = t5_1.aid)
+ -> Hash Join
+ Hash Cond: (prt5_2.b = t5_2.bid)
+ -> Seq Scan on prt5_p1 prt5_2
+ -> Hash
+ -> Seq Scan on t5_2
+ Filter: (blabel ~~ '%cd%'::text)
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%ab%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_3.a = t5_1.aid)
+ -> Hash Join
+ Hash Cond: (prt5_3.b = t5_2.bid)
+ -> Seq Scan on prt5_p2 prt5_3
+ -> Hash
+ -> Seq Scan on t5_2
+ Filter: (blabel ~~ '%cd%'::text)
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%ab%'::text)
+(34 rows)
+
+-- unable to extract non-partitioned right relation
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 RIGHT JOIN t5_1 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+-----------------------------------------------
+ Hash Right Join
+ Hash Cond: (prt5.a = t5_1.aid)
+ Join Filter: (t5_1.alabel ~~ '%abc%'::text)
+ -> Append
+ -> Seq Scan on prt5_p0 prt5_1
+ -> Seq Scan on prt5_p1 prt5_2
+ -> Seq Scan on prt5_p2 prt5_3
+ -> Hash
+ -> Seq Scan on t5_1
+(9 rows)
+
+-- left side can be extracted, but no cost benefit
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 LEFT JOIN t5_1 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+-------------------------------------------------
+ Hash Left Join
+ Hash Cond: (prt5.a = t5_1.aid)
+ -> Append
+ -> Seq Scan on prt5_p0 prt5_1
+ -> Seq Scan on prt5_p1 prt5_2
+ -> Seq Scan on prt5_p2 prt5_3
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+(9 rows)
+
+-- validation of the results with/without asymmetric partition-wise join
+SELECT * INTO pg_temp.result01a
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+SELECT * INTO pg_temp.result02a
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+SET enable_partitionwise_join = off;
+SELECT * INTO pg_temp.result01b
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+SELECT * INTO pg_temp.result02b
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+SELECT * FROM pg_temp.result01a EXCEPT SELECT * FROM pg_temp.result01b;
+ hkey | a | b | aid | alabel
+------+---+---+-----+--------
+(0 rows)
+
+SELECT * FROM pg_temp.result02a EXCEPT SELECT * FROM pg_temp.result02b;
+ hkey | a | b | aid | alabel | bid | blabel
+------+---+---+-----+--------+-----+--------
+(0 rows)
+
+RESET max_parallel_workers_per_gather;
diff --git a/src/test/regress/sql/partition_join.sql b/src/test/regress/sql/partition_join.sql
index 575ba7b8d4..ebf9649508 100644
--- a/src/test/regress/sql/partition_join.sql
+++ b/src/test/regress/sql/partition_join.sql
@@ -453,3 +453,67 @@ ANALYZE prt2;
EXPLAIN (COSTS OFF)
SELECT t1.a, t1.c, t2.b, t2.c FROM prt1 t1, prt2 t2 WHERE t1.a = t2.b AND t1.b = 0 ORDER BY t1.a, t2.b;
+
+--
+-- For asymmetric partition-wise join
+--
+CREATE TABLE prt5 (hkey int, a int, b int) PARTITION BY HASH(hkey);
+CREATE TABLE prt5_p0 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 0);
+CREATE TABLE prt5_p1 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 1);
+CREATE TABLE prt5_p2 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 2);
+CREATE TABLE t5_1 (aid int, alabel text);
+CREATE TABLE t5_2 (bid int, blabel text);
+
+INSERT INTO prt5 (SELECT x, (1000.0 * random())::int,
+ (1000.0 * random())::int
+ FROM generate_series(1,1000000) x);
+INSERT INTO t5_1 (SELECT x, md5(x::text) FROM generate_series(-200, 800) x);
+INSERT INTO t5_2 (SELECT x, md5(x::text) FROM generate_series(-200, 800) x);
+
+VACUUM ANALYZE prt5;
+VACUUM ANALYZE t5_1;
+VACUUM ANALYZE t5_2;
+
+SET max_parallel_workers_per_gather = 0;
+EXPLAIN (COSTS OFF)
+SELECT *
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+
+EXPLAIN (COSTS OFF)
+SELECT *
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+
+-- unable to extract non-partitioned right relation
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 RIGHT JOIN t5_1 ON a = aid AND alabel like '%abc%';
+-- left side can be extracted, but no cost benefit
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 LEFT JOIN t5_1 ON a = aid AND alabel like '%abc%';
+
+-- validation of the results with/without asymmetric partition-wise join
+SELECT * INTO pg_temp.result01a
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+
+SELECT * INTO pg_temp.result02a
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+
+SET enable_partitionwise_join = off;
+
+SELECT * INTO pg_temp.result01b
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+
+SELECT * INTO pg_temp.result02b
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+
+SELECT * FROM pg_temp.result01a EXCEPT SELECT * FROM pg_temp.result01b;
+SELECT * FROM pg_temp.result02a EXCEPT SELECT * FROM pg_temp.result02b;
+
+RESET max_parallel_workers_per_gather;
+
+
^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: Asymmetric partition-wise JOIN
2019-08-12 06:03 Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-08-22 16:05 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-08-23 22:01 ` Re: Asymmetric partition-wise JOIN Thomas Munro <[email protected]>
2019-08-24 08:33 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-12-01 03:24 ` Re: Asymmetric partition-wise JOIN Michael Paquier <[email protected]>
2019-12-27 07:34 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
@ 2020-03-27 14:44 ` David Steele <[email protected]>
2 siblings, 0 replies; 28+ messages in thread
From: David Steele @ 2020-03-27 14:44 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: Kohei KaiGai <[email protected]>; pgsql-hackers
Hi Thomas,
On 12/27/19 2:34 AM, Kohei KaiGai wrote:
>
> This crash was reproduced on our environment also.
> It looks to me adjust_child_relids_multilevel() didn't expect a case
> when supplied 'relids'
> (partially) indicate normal and non-partitioned relation.
> It tries to build a new 'parent_relids' that is a set of
> appinfo->parent_relid related to the
> supplied 'child_relids'. However, bits in child_relids that indicate
> normal relations are
> unintentionally dropped here. Then, adjust_child_relids_multilevel()
> goes to an infinite
> recursion until stack limitation.
>
> The attached v2 fixed the problem, and regression test finished correctly.
Any thoughts on the new version of this patch?
Regards,
--
-David
[email protected]
^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: Asymmetric partition-wise JOIN
2019-08-12 06:03 Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-08-22 16:05 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-08-23 22:01 ` Re: Asymmetric partition-wise JOIN Thomas Munro <[email protected]>
2019-08-24 08:33 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-12-01 03:24 ` Re: Asymmetric partition-wise JOIN Michael Paquier <[email protected]>
2019-12-27 07:34 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
@ 2020-07-01 09:10 ` Daniel Gustafsson <[email protected]>
2020-08-21 06:02 ` Re: Asymmetric partition-wise JOIN Andrey V. Lepikhov <[email protected]>
2 siblings, 1 reply; 28+ messages in thread
From: Daniel Gustafsson @ 2020-07-01 09:10 UTC (permalink / raw)
To: Kohei KaiGai <[email protected]>; +Cc: Michael Paquier <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers
> On 27 Dec 2019, at 08:34, Kohei KaiGai <[email protected]> wrote:
> The attached v2 fixed the problem, and regression test finished correctly.
This patch no longer applies to HEAD, please submit an rebased version.
Marking the entry Waiting on Author in the meantime.
cheers ./daniel
^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: Asymmetric partition-wise JOIN
2019-08-12 06:03 Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-08-22 16:05 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-08-23 22:01 ` Re: Asymmetric partition-wise JOIN Thomas Munro <[email protected]>
2019-08-24 08:33 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-12-01 03:24 ` Re: Asymmetric partition-wise JOIN Michael Paquier <[email protected]>
2019-12-27 07:34 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2020-07-01 09:10 ` Re: Asymmetric partition-wise JOIN Daniel Gustafsson <[email protected]>
@ 2020-08-21 06:02 ` Andrey V. Lepikhov <[email protected]>
2020-08-25 09:12 ` Re: Asymmetric partition-wise JOIN Daniel Gustafsson <[email protected]>
2020-11-09 10:53 ` Re: Asymmetric partition-wise JOIN Anastasia Lubennikova <[email protected]>
0 siblings, 2 replies; 28+ messages in thread
From: Andrey V. Lepikhov @ 2020-08-21 06:02 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; Kohei KaiGai <[email protected]>; +Cc: Michael Paquier <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers
On 7/1/20 2:10 PM, Daniel Gustafsson wrote:
>> On 27 Dec 2019, at 08:34, Kohei KaiGai <[email protected]> wrote:
>
>> The attached v2 fixed the problem, and regression test finished correctly.
>
> This patch no longer applies to HEAD, please submit an rebased version.
> Marking the entry Waiting on Author in the meantime.
Rebased version of the patch on current master (d259afa736).
I rebased it because it is a base of my experimental feature than we
don't break partitionwise join of a relation with foreign partition and
a local relation if we have info that remote server has foreign table
link to the local relation (by analogy with shippable extensions).
Maybe mark as 'Needs review'?
--
regards,
Andrey Lepikhov
Postgres Professional
^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: Asymmetric partition-wise JOIN
2019-08-12 06:03 Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-08-22 16:05 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-08-23 22:01 ` Re: Asymmetric partition-wise JOIN Thomas Munro <[email protected]>
2019-08-24 08:33 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-12-01 03:24 ` Re: Asymmetric partition-wise JOIN Michael Paquier <[email protected]>
2019-12-27 07:34 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2020-07-01 09:10 ` Re: Asymmetric partition-wise JOIN Daniel Gustafsson <[email protected]>
2020-08-21 06:02 ` Re: Asymmetric partition-wise JOIN Andrey V. Lepikhov <[email protected]>
@ 2020-08-25 09:12 ` Daniel Gustafsson <[email protected]>
1 sibling, 0 replies; 28+ messages in thread
From: Daniel Gustafsson @ 2020-08-25 09:12 UTC (permalink / raw)
To: Andrey V. Lepikhov <[email protected]>; +Cc: Kohei KaiGai <[email protected]>; Michael Paquier <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers
> On 21 Aug 2020, at 08:02, Andrey V. Lepikhov <[email protected]> wrote:
>
> On 7/1/20 2:10 PM, Daniel Gustafsson wrote:
>>> On 27 Dec 2019, at 08:34, Kohei KaiGai <[email protected]> wrote:
>>> The attached v2 fixed the problem, and regression test finished correctly.
>> This patch no longer applies to HEAD, please submit an rebased version.
>> Marking the entry Waiting on Author in the meantime.
> Rebased version of the patch on current master (d259afa736).
>
> I rebased it because it is a base of my experimental feature than we don't break partitionwise join of a relation with foreign partition and a local relation if we have info that remote server has foreign table link to the local relation (by analogy with shippable extensions).
>
> Maybe mark as 'Needs review'?
Thanks for the rebase, I've updated the commitfest entry to reflect that it
needs a round of review.
cheers ./daniel
^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: Asymmetric partition-wise JOIN
2019-08-12 06:03 Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-08-22 16:05 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-08-23 22:01 ` Re: Asymmetric partition-wise JOIN Thomas Munro <[email protected]>
2019-08-24 08:33 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-12-01 03:24 ` Re: Asymmetric partition-wise JOIN Michael Paquier <[email protected]>
2019-12-27 07:34 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2020-07-01 09:10 ` Re: Asymmetric partition-wise JOIN Daniel Gustafsson <[email protected]>
2020-08-21 06:02 ` Re: Asymmetric partition-wise JOIN Andrey V. Lepikhov <[email protected]>
@ 2020-11-09 10:53 ` Anastasia Lubennikova <[email protected]>
2020-11-30 14:43 ` Re: Asymmetric partition-wise JOIN Anastasia Lubennikova <[email protected]>
1 sibling, 1 reply; 28+ messages in thread
From: Anastasia Lubennikova @ 2020-11-09 10:53 UTC (permalink / raw)
To: Andrey V. Lepikhov <[email protected]>; Daniel Gustafsson <[email protected]>; Kohei KaiGai <[email protected]>; +Cc: Michael Paquier <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers
On 21.08.2020 09:02, Andrey V. Lepikhov wrote:
> On 7/1/20 2:10 PM, Daniel Gustafsson wrote:
>>> On 27 Dec 2019, at 08:34, Kohei KaiGai <[email protected]> wrote:
>>
>>> The attached v2 fixed the problem, and regression test finished
>>> correctly.
>>
>> This patch no longer applies to HEAD, please submit an rebased version.
>> Marking the entry Waiting on Author in the meantime.
> Rebased version of the patch on current master (d259afa736).
>
> I rebased it because it is a base of my experimental feature than we
> don't break partitionwise join of a relation with foreign partition
> and a local relation if we have info that remote server has foreign
> table link to the local relation (by analogy with shippable extensions).
>
> Maybe mark as 'Needs review'?
>
Status update for a commitfest entry.
According to cfbot, the patch fails to apply. Could you please send a
rebased version?
This thread was inactive for quite some time. Is anyone going to
continue working on it?
I see some interest in the idea of sharable hash, but I don't see even a
prototype in this thread. So, probably, it is a matter of a separate
discussion.
Also, I took a look at the code. It looks like it needs some extra work.
I am not a big expert in this area, so I'm sorry if questions are obvious.
1. What would happen if this assumption is not met?
+ * MEMO: We assume this pathlist keeps at least one AppendPath that
+ * represents partitioned table-scan, symmetric or asymmetric
+ * partition-wise join. It is not correct right now, however, a
hook
+ * on add_path() to give additional decision for path removel
allows
+ * to retain this kind of AppendPath, regardless of its cost.
2. Why do we wrap extract_asymmetric_partitionwise_subjoin() call into
PG_TRY/PG_CATCH? What errors do we expect?
3. It looks like a crutch. If it isn't, I'd like to see a better comment
about why "dynamic programming" is not applicable here.
And shouldn't we also handle a root->join_cur_level?
+ /* temporary disables "dynamic programming" algorithm */
+ root->join_rel_level = NULL;
4. This change looks like it can lead to a memory leak for old code.
Maybe it is never the case, but again I think it worth a comment.
- /* If there's nothing to adjust, don't call this function. */
- Assert(nappinfos >= 1 && appinfos != NULL);
+ /* If there's nothing to adjust, just return a duplication */
+ if (nappinfos == 0)
+ return copyObject(node);
5. extract_asymmetric_partitionwise_subjoin() lacks a comment
The new status of this patch is: Waiting on Author
--
Anastasia Lubennikova
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: Asymmetric partition-wise JOIN
2019-08-12 06:03 Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-08-22 16:05 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-08-23 22:01 ` Re: Asymmetric partition-wise JOIN Thomas Munro <[email protected]>
2019-08-24 08:33 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-12-01 03:24 ` Re: Asymmetric partition-wise JOIN Michael Paquier <[email protected]>
2019-12-27 07:34 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2020-07-01 09:10 ` Re: Asymmetric partition-wise JOIN Daniel Gustafsson <[email protected]>
2020-08-21 06:02 ` Re: Asymmetric partition-wise JOIN Andrey V. Lepikhov <[email protected]>
2020-11-09 10:53 ` Re: Asymmetric partition-wise JOIN Anastasia Lubennikova <[email protected]>
@ 2020-11-30 14:43 ` Anastasia Lubennikova <[email protected]>
2021-04-09 11:14 ` Re: Asymmetric partition-wise JOIN Andrey V. Lepikhov <[email protected]>
2021-04-30 03:10 ` Re: Asymmetric partition-wise JOIN Andrey V. Lepikhov <[email protected]>
0 siblings, 2 replies; 28+ messages in thread
From: Anastasia Lubennikova @ 2020-11-30 14:43 UTC (permalink / raw)
To: Andrey V. Lepikhov <[email protected]>; Daniel Gustafsson <[email protected]>; Kohei KaiGai <[email protected]>; +Cc: Michael Paquier <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers
On 09.11.2020 13:53, Anastasia Lubennikova wrote:
> On 21.08.2020 09:02, Andrey V. Lepikhov wrote:
>> On 7/1/20 2:10 PM, Daniel Gustafsson wrote:
>>>> On 27 Dec 2019, at 08:34, Kohei KaiGai <[email protected]> wrote:
>>>
>>>> The attached v2 fixed the problem, and regression test finished
>>>> correctly.
>>>
>>> This patch no longer applies to HEAD, please submit an rebased version.
>>> Marking the entry Waiting on Author in the meantime.
>> Rebased version of the patch on current master (d259afa736).
>>
>> I rebased it because it is a base of my experimental feature than we
>> don't break partitionwise join of a relation with foreign partition
>> and a local relation if we have info that remote server has foreign
>> table link to the local relation (by analogy with shippable extensions).
>>
>> Maybe mark as 'Needs review'?
>>
> Status update for a commitfest entry.
>
> According to cfbot, the patch fails to apply. Could you please send a
> rebased version?
>
> This thread was inactive for quite some time. Is anyone going to
> continue working on it?
>
> I see some interest in the idea of sharable hash, but I don't see even
> a prototype in this thread. So, probably, it is a matter of a separate
> discussion.
>
> Also, I took a look at the code. It looks like it needs some extra
> work. I am not a big expert in this area, so I'm sorry if questions
> are obvious.
>
> 1. What would happen if this assumption is not met?
>
> + * MEMO: We assume this pathlist keeps at least one
> AppendPath that
> + * represents partitioned table-scan, symmetric or asymmetric
> + * partition-wise join. It is not correct right now, however,
> a hook
> + * on add_path() to give additional decision for path removel
> allows
> + * to retain this kind of AppendPath, regardless of its cost.
>
> 2. Why do we wrap extract_asymmetric_partitionwise_subjoin() call into
> PG_TRY/PG_CATCH? What errors do we expect?
>
> 3. It looks like a crutch. If it isn't, I'd like to see a better
> comment about why "dynamic programming" is not applicable here.
> And shouldn't we also handle a root->join_cur_level?
>
> + /* temporary disables "dynamic programming" algorithm */
> + root->join_rel_level = NULL;
>
> 4. This change looks like it can lead to a memory leak for old code.
> Maybe it is never the case, but again I think it worth a comment.
>
> - /* If there's nothing to adjust, don't call this function. */
> - Assert(nappinfos >= 1 && appinfos != NULL);
> + /* If there's nothing to adjust, just return a duplication */
> + if (nappinfos == 0)
> + return copyObject(node);
>
> 5. extract_asymmetric_partitionwise_subjoin() lacks a comment
>
> The new status of this patch is: Waiting on Author
>
Status update for a commitfest entry.
This entry was inactive during this CF, so I've marked it as returned
with feedback. Feel free to resubmit an updated version to a future
commitfest.
--
Anastasia Lubennikova
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: Asymmetric partition-wise JOIN
2019-08-12 06:03 Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-08-22 16:05 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-08-23 22:01 ` Re: Asymmetric partition-wise JOIN Thomas Munro <[email protected]>
2019-08-24 08:33 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-12-01 03:24 ` Re: Asymmetric partition-wise JOIN Michael Paquier <[email protected]>
2019-12-27 07:34 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2020-07-01 09:10 ` Re: Asymmetric partition-wise JOIN Daniel Gustafsson <[email protected]>
2020-08-21 06:02 ` Re: Asymmetric partition-wise JOIN Andrey V. Lepikhov <[email protected]>
2020-11-09 10:53 ` Re: Asymmetric partition-wise JOIN Anastasia Lubennikova <[email protected]>
2020-11-30 14:43 ` Re: Asymmetric partition-wise JOIN Anastasia Lubennikova <[email protected]>
@ 2021-04-09 11:14 ` Andrey V. Lepikhov <[email protected]>
1 sibling, 0 replies; 28+ messages in thread
From: Andrey V. Lepikhov @ 2021-04-09 11:14 UTC (permalink / raw)
To: Anastasia Lubennikova <[email protected]>; Daniel Gustafsson <[email protected]>; Kohei KaiGai <[email protected]>; +Cc: Michael Paquier <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers
On 11/30/20 7:43 PM, Anastasia Lubennikova wrote:
> This entry was inactive during this CF, so I've marked it as returned
> with feedback. Feel free to resubmit an updated version to a future
> commitfest.
>
Attached version is rebased on current master and fixes problems with
complex parameterized plans - 'reparameterize by child' feature.
Problems with reparameterization machinery can be demonstrated by TPC-H
benchmark.
--
regards,
Andrey Lepikhov
Postgres Professional
^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: Asymmetric partition-wise JOIN
2019-08-12 06:03 Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-08-22 16:05 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-08-23 22:01 ` Re: Asymmetric partition-wise JOIN Thomas Munro <[email protected]>
2019-08-24 08:33 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-12-01 03:24 ` Re: Asymmetric partition-wise JOIN Michael Paquier <[email protected]>
2019-12-27 07:34 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2020-07-01 09:10 ` Re: Asymmetric partition-wise JOIN Daniel Gustafsson <[email protected]>
2020-08-21 06:02 ` Re: Asymmetric partition-wise JOIN Andrey V. Lepikhov <[email protected]>
2020-11-09 10:53 ` Re: Asymmetric partition-wise JOIN Anastasia Lubennikova <[email protected]>
2020-11-30 14:43 ` Re: Asymmetric partition-wise JOIN Anastasia Lubennikova <[email protected]>
@ 2021-04-30 03:10 ` Andrey V. Lepikhov <[email protected]>
2021-05-27 04:27 ` Re: Asymmetric partition-wise JOIN Andrey Lepikhov <[email protected]>
1 sibling, 1 reply; 28+ messages in thread
From: Andrey V. Lepikhov @ 2021-04-30 03:10 UTC (permalink / raw)
To: Anastasia Lubennikova <[email protected]>; Daniel Gustafsson <[email protected]>; Kohei KaiGai <[email protected]>; +Cc: Michael Paquier <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers
On 11/30/20 7:43 PM, Anastasia Lubennikova wrote:
> This entry was inactive during this CF, so I've marked it as returned
> with feedback. Feel free to resubmit an updated version to a future
> commitfest.
I return the patch to commitfest. My current reason differs from reason
of origin author.
This patch can open a door for more complex optimizations in the
partitionwise join push-down technique.
I mean, we can push-down join not only of two partitioned tables with
the same partition schema, but a partitioned (sharded) table with an
arbitrary subplan that is provable independent of local resources.
Example:
CREATE TABLE p(a int) PARTITION BY HASH (a);
CREATE TABLE p1 PARTITION OF p FOR VALUES WITH (MODULUS 3, REMAINDER 0);
CREATE TABLE p2 PARTITION OF p FOR VALUES WITH (MODULUS 3, REMAINDER 1);
CREATE TABLE p3 PARTITION OF p FOR VALUES WITH (MODULUS 3, REMAINDER 2);
SELECT * FROM p, (SELECT * FROM generate_series(1,2) AS a) AS s
WHERE p.a=s.a;
Hash Join
Hash Cond: (p.a = a.a)
-> Append
-> Seq Scan on p1 p_1
-> Seq Scan on p2 p_2
-> Seq Scan on p3 p_3
-> Hash
-> Function Scan on generate_series a
But with asymmetric join feature we have the plan:
Append
-> Hash Join
Hash Cond: (p_1.a = a.a)
-> Seq Scan on p1 p_1
-> Hash
-> Function Scan on generate_series a
-> Hash Join
Hash Cond: (p_2.a = a.a)
-> Seq Scan on p2 p_2
-> Hash
-> Function Scan on generate_series a
-> Hash Join
Hash Cond: (p_3.a = a.a)
-> Seq Scan on p3 p_3
-> Hash
-> Function Scan on generate_series a
In the case of FDW-sharding it means that if we can prove that the inner
relation is independent from the execution server, we can push-down
these joins and execute it in parallel.
--
regards,
Andrey Lepikhov
Postgres Professional
^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: Asymmetric partition-wise JOIN
2019-08-12 06:03 Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-08-22 16:05 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-08-23 22:01 ` Re: Asymmetric partition-wise JOIN Thomas Munro <[email protected]>
2019-08-24 08:33 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-12-01 03:24 ` Re: Asymmetric partition-wise JOIN Michael Paquier <[email protected]>
2019-12-27 07:34 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2020-07-01 09:10 ` Re: Asymmetric partition-wise JOIN Daniel Gustafsson <[email protected]>
2020-08-21 06:02 ` Re: Asymmetric partition-wise JOIN Andrey V. Lepikhov <[email protected]>
2020-11-09 10:53 ` Re: Asymmetric partition-wise JOIN Anastasia Lubennikova <[email protected]>
2020-11-30 14:43 ` Re: Asymmetric partition-wise JOIN Anastasia Lubennikova <[email protected]>
2021-04-30 03:10 ` Re: Asymmetric partition-wise JOIN Andrey V. Lepikhov <[email protected]>
@ 2021-05-27 04:27 ` Andrey Lepikhov <[email protected]>
2021-06-18 12:02 ` Re: Asymmetric partition-wise JOIN Alexander Pyhalov <[email protected]>
0 siblings, 1 reply; 28+ messages in thread
From: Andrey Lepikhov @ 2021-05-27 04:27 UTC (permalink / raw)
To: Anastasia Lubennikova <[email protected]>; Daniel Gustafsson <[email protected]>; Kohei KaiGai <[email protected]>; +Cc: Michael Paquier <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers
Next version of the patch.
For searching any problems I forced this patch during 'make check'
tests. Some bugs were found and fixed.
--
regards,
Andrey Lepikhov
Postgres Professional
From 101614b504b0b17e201d2375c8af61cfc671e51d Mon Sep 17 00:00:00 2001
From: Andrey Lepikhov <[email protected]>
Date: Fri, 2 Apr 2021 11:02:20 +0500
Subject: [PATCH] Asymmetric partitionwise join.
Teach optimizer to consider partitionwise join of non-partitioned
table with each partition of partitioned table.
Disallow asymmetric machinery for joining of two partitioned (or appended)
relations because it could cause CPU and memory huge consumption
during reparameterization of NestLoop path.
---
src/backend/optimizer/path/joinpath.c | 9 +
src/backend/optimizer/path/joinrels.c | 184 +++++++++++
src/backend/optimizer/plan/setrefs.c | 13 +-
src/backend/optimizer/util/appendinfo.c | 37 ++-
src/backend/optimizer/util/pathnode.c | 9 +-
src/backend/optimizer/util/relnode.c | 19 +-
src/include/optimizer/paths.h | 7 +-
src/test/regress/expected/partition_join.out | 306 +++++++++++++++++++
src/test/regress/sql/partition_join.sql | 126 ++++++++
9 files changed, 682 insertions(+), 28 deletions(-)
diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c
index b67b517770..7a1a7b2e86 100644
--- a/src/backend/optimizer/path/joinpath.c
+++ b/src/backend/optimizer/path/joinpath.c
@@ -335,6 +335,15 @@ add_paths_to_joinrel(PlannerInfo *root,
if (set_join_pathlist_hook)
set_join_pathlist_hook(root, joinrel, outerrel, innerrel,
jointype, &extra);
+
+ /*
+ * 7. If outer relation is delivered from partition-tables, consider
+ * distributing inner relation to every partition-leaf prior to
+ * append these leafs.
+ */
+ try_asymmetric_partitionwise_join(root, joinrel,
+ outerrel, innerrel,
+ jointype, &extra);
}
/*
diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c
index 0dbe2ac726..e6bd2ea5fe 100644
--- a/src/backend/optimizer/path/joinrels.c
+++ b/src/backend/optimizer/path/joinrels.c
@@ -16,6 +16,7 @@
#include "miscadmin.h"
#include "optimizer/appendinfo.h"
+#include "optimizer/cost.h"
#include "optimizer/joininfo.h"
#include "optimizer/pathnode.h"
#include "optimizer/paths.h"
@@ -1551,6 +1552,189 @@ try_partitionwise_join(PlannerInfo *root, RelOptInfo *rel1, RelOptInfo *rel2,
}
}
+/*
+ * Build RelOptInfo on JOIN of each partition of the outer relation and the inner
+ * relation. Return List of such RelOptInfo's. Return NIL, if at least one of
+ * these JOINs are impossible to build.
+ */
+static List *
+extract_asymmetric_partitionwise_subjoin(PlannerInfo *root,
+ RelOptInfo *joinrel,
+ AppendPath *append_path,
+ RelOptInfo *inner_rel,
+ JoinType jointype,
+ JoinPathExtraData *extra)
+{
+ List *result = NIL;
+ ListCell *lc;
+
+ foreach (lc, append_path->subpaths)
+ {
+ Path *child_path = lfirst(lc);
+ RelOptInfo *child_rel = child_path->parent;
+ Relids child_join_relids;
+ Relids parent_relids;
+ RelOptInfo *child_join_rel;
+ SpecialJoinInfo *child_sjinfo;
+ List *child_restrictlist;
+
+ child_join_relids = bms_union(child_rel->relids, inner_rel->relids);
+ parent_relids = bms_union(append_path->path.parent->relids,
+ inner_rel->relids);
+
+ child_sjinfo = build_child_join_sjinfo(root, extra->sjinfo,
+ child_rel->relids,
+ inner_rel->relids);
+ child_restrictlist = (List *)
+ adjust_appendrel_attrs_multilevel(root, (Node *)extra->restrictlist,
+ child_join_relids, parent_relids);
+
+ child_join_rel = find_join_rel(root, child_join_relids);
+ if (!child_join_rel)
+ {
+ child_join_rel = build_child_join_rel(root,
+ child_rel,
+ inner_rel,
+ joinrel,
+ child_restrictlist,
+ child_sjinfo,
+ jointype);
+ if (!child_join_rel)
+ {
+ /*
+ * If can't build JOIN between inner relation and one of the outer
+ * partitions - return immediately.
+ */
+ return NIL;
+ }
+ }
+ else
+ {
+ /*
+ * TODO:
+ * Can't imagine situation when join relation already exists. But in
+ * the 'partition_join' regression test it happens.
+ * It may be an indicator of possible problems.
+ */
+ }
+
+ populate_joinrel_with_paths(root,
+ child_rel,
+ inner_rel,
+ child_join_rel,
+ child_sjinfo,
+ child_restrictlist);
+
+ /* Give up if asymmetric partition-wise join is not available */
+ if (child_join_rel->pathlist == NIL)
+ return NIL;
+
+ set_cheapest(child_join_rel);
+ result = lappend(result, child_join_rel);
+ }
+ return result;
+}
+
+static bool
+is_asymmetric_join_capable(PlannerInfo *root,
+ RelOptInfo *outer_rel,
+ RelOptInfo *inner_rel,
+ JoinType jointype)
+{
+ ListCell *lc;
+
+ if (jointype != JOIN_INNER && jointype != JOIN_LEFT)
+ return false;
+
+ if (IS_OTHER_REL(outer_rel) || IS_OTHER_REL(inner_rel))
+ return false;
+
+ /* Disallow recursive usage of asymmertic join machinery */
+ if (root->join_rel_level == NULL)
+ return false;
+
+ /*
+ * Don't allow asymmetric JOIN of two append subplans.
+ * In the case of a parameterized NL join, a reparameterization procedure will
+ * lead to large memory allocations and a CPU consumption:
+ * each reparameterize will induce subpath duplication, creating new
+ * ParamPathInfo instance and increasing of ppilist up to number of partitions
+ * in the inner. Also, if we have many partitions, each bitmapset
+ * variable will large and many leaks of such variable (caused by relid
+ * replacement) will highly increase memory consumption.
+ * So, we deny such paths for now.
+ */
+ foreach(lc, inner_rel->pathlist)
+ {
+ if (IsA(lfirst(lc), AppendPath))
+ return false;
+ }
+
+ return true;
+}
+
+void
+try_asymmetric_partitionwise_join(PlannerInfo *root,
+ RelOptInfo *joinrel,
+ RelOptInfo *outer_rel,
+ RelOptInfo *inner_rel,
+ JoinType jointype,
+ JoinPathExtraData *extra)
+{
+ ListCell *lc;
+
+ /*
+ * Try this kind of paths if we allow complex partitionwise joins and we know
+ * we can build this join safely.
+ */
+ if (!enable_partitionwise_join ||
+ !is_asymmetric_join_capable(root, outer_rel, inner_rel, jointype))
+ return;
+
+ foreach (lc, outer_rel->pathlist)
+ {
+ AppendPath *append_path = lfirst(lc);
+
+ /*
+ * MEMO: We assume this pathlist keeps at least one AppendPath that
+ * represents partitioned table-scan, symmetric or asymmetric
+ * partition-wise join. It is not correct right now, however, a hook
+ * on add_path() to give additional decision for path removal allows
+ * to retain this kind of AppendPath, regardless of its cost.
+ */
+ if (IsA(append_path, AppendPath))
+ {
+ List **join_rel_level_saved;
+ List *live_childrels = NIL;
+
+ join_rel_level_saved = root->join_rel_level;
+ PG_TRY();
+ {
+ /* temporary disables "dynamic programming" algorithm */
+ root->join_rel_level = NULL;
+
+ live_childrels =
+ extract_asymmetric_partitionwise_subjoin(root,
+ joinrel,
+ append_path,
+ inner_rel,
+ jointype,
+ extra);
+ }
+ PG_FINALLY();
+ {
+ root->join_rel_level = join_rel_level_saved;
+ }
+ PG_END_TRY();
+
+ if (live_childrels != NIL)
+ add_paths_to_append_rel(root, joinrel, live_childrels);
+
+ break;
+ }
+ }
+}
+
/*
* Construct the SpecialJoinInfo for a child-join by translating
* SpecialJoinInfo for the join between parents. left_relids and right_relids
diff --git a/src/backend/optimizer/plan/setrefs.c b/src/backend/optimizer/plan/setrefs.c
index 61ccfd300b..dbe568aa29 100644
--- a/src/backend/optimizer/plan/setrefs.c
+++ b/src/backend/optimizer/plan/setrefs.c
@@ -287,18 +287,23 @@ set_plan_references(PlannerInfo *root, Plan *plan)
foreach(lc, root->append_rel_list)
{
AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc);
+ AppendRelInfo *newappinfo;
+
+ /* flat copy is enough since all valuable fields are scalars */
+ newappinfo = (AppendRelInfo *) palloc(sizeof(AppendRelInfo));
+ memcpy(newappinfo, appinfo, sizeof(AppendRelInfo));
/* adjust RT indexes */
- appinfo->parent_relid += rtoffset;
- appinfo->child_relid += rtoffset;
+ newappinfo->parent_relid += rtoffset;
+ newappinfo->child_relid += rtoffset;
/*
* Rather than adjust the translated_vars entries, just drop 'em.
* Neither the executor nor EXPLAIN currently need that data.
*/
- appinfo->translated_vars = NIL;
+ newappinfo->translated_vars = NIL;
- glob->appendRelations = lappend(glob->appendRelations, appinfo);
+ glob->appendRelations = lappend(glob->appendRelations, newappinfo);
}
/* Now fix the Plan tree */
diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c
index af46f581ac..ec8dee99b0 100644
--- a/src/backend/optimizer/util/appendinfo.c
+++ b/src/backend/optimizer/util/appendinfo.c
@@ -202,7 +202,9 @@ adjust_appendrel_attrs(PlannerInfo *root, Node *node, int nappinfos,
context.appinfos = appinfos;
/* If there's nothing to adjust, don't call this function. */
- Assert(nappinfos >= 1 && appinfos != NULL);
+ /* If there's nothing to adjust, just return a duplication */
+ if (nappinfos == 0)
+ return copyObject(node);
/* Should never be translating a Query tree. */
Assert(node == NULL || !IsA(node, Query));
@@ -489,12 +491,10 @@ adjust_appendrel_attrs_multilevel(PlannerInfo *root, Node *node,
Relids child_relids,
Relids top_parent_relids)
{
- AppendRelInfo **appinfos;
- Bitmapset *parent_relids = NULL;
- int nappinfos;
- int cnt;
-
- Assert(bms_num_members(child_relids) == bms_num_members(top_parent_relids));
+ AppendRelInfo **appinfos;
+ Relids parent_relids = NULL;
+ int nappinfos;
+ int cnt;
appinfos = find_appinfos_by_relids(root, child_relids, &nappinfos);
@@ -506,8 +506,13 @@ adjust_appendrel_attrs_multilevel(PlannerInfo *root, Node *node,
parent_relids = bms_add_member(parent_relids, appinfo->parent_relid);
}
- /* Recurse if immediate parent is not the top parent. */
- if (!bms_equal(parent_relids, top_parent_relids))
+ /*
+ * Recurse if immediate parent is not the top parent. Keep in mind that in a
+ * case of asymmetric JOIN top_parent_relids can contain relids which aren't
+ * part of an append node.
+ */
+ if (!bms_equal(parent_relids, top_parent_relids) &&
+ !bms_is_subset(parent_relids, top_parent_relids))
node = adjust_appendrel_attrs_multilevel(root, node, parent_relids,
top_parent_relids);
@@ -515,6 +520,7 @@ adjust_appendrel_attrs_multilevel(PlannerInfo *root, Node *node,
node = adjust_appendrel_attrs(root, node, nappinfos, appinfos);
pfree(appinfos);
+ pfree(parent_relids);
return node;
}
@@ -565,6 +571,7 @@ adjust_child_relids_multilevel(PlannerInfo *root, Relids relids,
AppendRelInfo **appinfos;
int nappinfos;
Relids parent_relids = NULL;
+ Relids normal_relids = NULL;
Relids result;
Relids tmp_result = NULL;
int cnt;
@@ -579,12 +586,17 @@ adjust_child_relids_multilevel(PlannerInfo *root, Relids relids,
appinfos = find_appinfos_by_relids(root, child_relids, &nappinfos);
/* Construct relids set for the immediate parent of the given child. */
+ normal_relids = bms_copy(child_relids);
for (cnt = 0; cnt < nappinfos; cnt++)
{
AppendRelInfo *appinfo = appinfos[cnt];
parent_relids = bms_add_member(parent_relids, appinfo->parent_relid);
+ normal_relids = bms_del_member(normal_relids, appinfo->child_relid);
}
+ parent_relids = bms_union(parent_relids, normal_relids);
+ if (normal_relids)
+ bms_free(normal_relids);
/* Recurse if immediate parent is not the top parent. */
if (!bms_equal(parent_relids, top_parent_relids))
@@ -715,11 +727,11 @@ AppendRelInfo **
find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos)
{
AppendRelInfo **appinfos;
+ int nrooms = bms_num_members(relids);
int cnt = 0;
int i;
- *nappinfos = bms_num_members(relids);
- appinfos = (AppendRelInfo **) palloc(sizeof(AppendRelInfo *) * *nappinfos);
+ appinfos = (AppendRelInfo **) palloc(sizeof(AppendRelInfo *) * nrooms);
i = -1;
while ((i = bms_next_member(relids, i)) >= 0)
@@ -727,10 +739,11 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos)
AppendRelInfo *appinfo = root->append_rel_array[i];
if (!appinfo)
- elog(ERROR, "child rel %d not found in append_rel_array", i);
+ continue;
appinfos[cnt++] = appinfo;
}
+ *nappinfos = cnt;
return appinfos;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index b248b038e0..73cfb4748d 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -4209,7 +4209,14 @@ do { \
MemoryContextSwitchTo(oldcontext);
}
- bms_free(required_outer);
+
+ /*
+ * If adjust_child_relids_multilevel don't do replacements it returns
+ * the original set, not a copy. It is possible in the case of asymmetric
+ * JOIN and child_rel->relids contains relids only of plane relations.
+ */
+ if (required_outer != old_ppi->ppi_req_outer)
+ bms_free(required_outer);
new_path->param_info = new_ppi;
diff --git a/src/backend/optimizer/util/relnode.c b/src/backend/optimizer/util/relnode.c
index e105a4d5f1..bb15fb2716 100644
--- a/src/backend/optimizer/util/relnode.c
+++ b/src/backend/optimizer/util/relnode.c
@@ -790,11 +790,8 @@ build_child_join_rel(PlannerInfo *root, RelOptInfo *outer_rel,
AppendRelInfo **appinfos;
int nappinfos;
- /* Only joins between "other" relations land here. */
- Assert(IS_OTHER_REL(outer_rel) && IS_OTHER_REL(inner_rel));
-
- /* The parent joinrel should have consider_partitionwise_join set. */
- Assert(parent_joinrel->consider_partitionwise_join);
+ /* Either of relations must be "other" relation at least. */
+ Assert(IS_OTHER_REL(outer_rel) || IS_OTHER_REL(inner_rel));
joinrel->reloptkind = RELOPT_OTHER_JOINREL;
joinrel->relids = bms_union(outer_rel->relids, inner_rel->relids);
@@ -851,8 +848,11 @@ build_child_join_rel(PlannerInfo *root, RelOptInfo *outer_rel,
joinrel->partexprs = NULL;
joinrel->nullable_partexprs = NULL;
- joinrel->top_parent_relids = bms_union(outer_rel->top_parent_relids,
- inner_rel->top_parent_relids);
+ joinrel->top_parent_relids =
+ bms_union(IS_OTHER_REL(outer_rel) ?
+ outer_rel->top_parent_relids : outer_rel->relids,
+ IS_OTHER_REL(inner_rel) ?
+ inner_rel->top_parent_relids : inner_rel->relids);
/* Compute information relevant to foreign relations. */
set_foreign_rel_properties(joinrel, outer_rel, inner_rel);
@@ -2033,9 +2033,8 @@ build_child_join_reltarget(PlannerInfo *root,
{
/* Build the targetlist */
childrel->reltarget->exprs = (List *)
- adjust_appendrel_attrs(root,
- (Node *) parentrel->reltarget->exprs,
- nappinfos, appinfos);
+ adjust_appendrel_attrs_multilevel(root, (Node *)parentrel->reltarget->exprs,
+ childrel->relids, parentrel->relids);
/* Set the cost and width fields */
childrel->reltarget->cost.startup = parentrel->reltarget->cost.startup;
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index f1d111063c..a0106dc539 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -109,7 +109,12 @@ extern bool have_join_order_restriction(PlannerInfo *root,
extern bool have_dangerous_phv(PlannerInfo *root,
Relids outer_relids, Relids inner_params);
extern void mark_dummy_rel(RelOptInfo *rel);
-
+extern void try_asymmetric_partitionwise_join(PlannerInfo *root,
+ RelOptInfo *joinrel,
+ RelOptInfo *outer_rel,
+ RelOptInfo *inner_rel,
+ JoinType jointype,
+ JoinPathExtraData *extra);
/*
* equivclass.c
* routines for managing EquivalenceClasses
diff --git a/src/test/regress/expected/partition_join.out b/src/test/regress/expected/partition_join.out
index 27f7525b3e..601f08662e 100644
--- a/src/test/regress/expected/partition_join.out
+++ b/src/test/regress/expected/partition_join.out
@@ -2320,6 +2320,312 @@ SELECT t1.a, t1.c, t2.b, t2.c FROM prt1_adv t1 INNER JOIN prt2_adv t2 ON (t1.a =
375 | 0375 | 375 | 0375
(8 rows)
+--
+-- For asymmetric partition-wise join
+--
+CREATE TABLE prt5 (hkey int, a int, b int) PARTITION BY HASH(hkey);
+CREATE TABLE prt5_p0 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 0);
+CREATE TABLE prt5_p1 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 1);
+CREATE TABLE prt5_p2 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 2);
+CREATE TABLE prt6 (aid int, alabel text) PARTITION BY HASH(aid);
+CREATE TABLE prt6_p0 PARTITION OF prt6
+ FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE prt6_p1 PARTITION OF prt6
+ FOR VALUES WITH (modulus 2, remainder 1);
+CREATE TABLE t5_1 (aid int, alabel text);
+CREATE TABLE t5_2 (bid int, blabel text);
+INSERT INTO prt5 (SELECT x, (1000.0 * random())::int,
+ (1000.0 * random())::int
+ FROM generate_series(1,1000000) x);
+INSERT INTO t5_1 (SELECT x, md5(x::text) FROM generate_series(-200, 800) x);
+INSERT INTO t5_2 (SELECT x, md5(x::text) FROM generate_series(-200, 800) x);
+INSERT INTO prt6 (SELECT * FROM t5_1);
+VACUUM ANALYZE prt5,prt6,t5_1,t5_2;
+SET max_parallel_workers_per_gather = 0;
+-- Trivial asymmetric JOIN of partitioned table with a relation
+EXPLAIN (COSTS OFF)
+SELECT *
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+-------------------------------------------------------
+ Append
+ -> Hash Join
+ Hash Cond: (prt5_1.a = t5_1.aid)
+ -> Seq Scan on prt5_p0 prt5_1
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_2.a = t5_1.aid)
+ -> Seq Scan on prt5_p1 prt5_2
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_3.a = t5_1.aid)
+ -> Seq Scan on prt5_p2 prt5_3
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+(19 rows)
+
+-- The same, but appended with UNION ALL
+EXPLAIN (COSTS OFF)
+SELECT * FROM (
+ (SELECT * FROM prt5_p0)
+ UNION ALL
+ (SELECT * FROM prt5_p1)
+ UNION ALL
+ (SELECT * FROM prt5_p2)
+ ) AS sq1
+JOIN t5_1 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+-------------------------------------------------------
+ Append
+ -> Hash Join
+ Hash Cond: (prt5_p0.a = t5_1.aid)
+ -> Seq Scan on prt5_p0
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_p1.a = t5_1.aid)
+ -> Seq Scan on prt5_p1
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_p2.a = t5_1.aid)
+ -> Seq Scan on prt5_p2
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+(19 rows)
+
+-- Don't allow asymmetric JOIN of two partitioned tables.
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 JOIN prt6 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+-------------------------------------------------------
+ Hash Join
+ Hash Cond: (prt5.a = prt6.aid)
+ -> Append
+ -> Seq Scan on prt5_p0 prt5_1
+ -> Seq Scan on prt5_p1 prt5_2
+ -> Seq Scan on prt5_p2 prt5_3
+ -> Hash
+ -> Append
+ -> Seq Scan on prt6_p0 prt6_1
+ Filter: (alabel ~~ '%abc%'::text)
+ -> Seq Scan on prt6_p1 prt6_2
+ Filter: (alabel ~~ '%abc%'::text)
+(12 rows)
+
+-- Check asymmetric JOIN with Subquery
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 JOIN (
+ SELECT * FROM prt6 LIMIT 1000
+) AS sq1 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+----------------------------------------------------------------
+ Append
+ -> Hash Join
+ Hash Cond: (prt5_1.a = sq1.aid)
+ -> Seq Scan on prt5_p0 prt5_1
+ -> Hash
+ -> Subquery Scan on sq1
+ Filter: (sq1.alabel ~~ '%abc%'::text)
+ -> Limit
+ -> Append
+ -> Seq Scan on prt6_p0 prt6_1
+ -> Seq Scan on prt6_p1 prt6_2
+ -> Hash Join
+ Hash Cond: (prt5_2.a = sq1.aid)
+ -> Seq Scan on prt5_p1 prt5_2
+ -> Hash
+ -> Subquery Scan on sq1
+ Filter: (sq1.alabel ~~ '%abc%'::text)
+ -> Limit
+ -> Append
+ -> Seq Scan on prt6_p0 prt6_4
+ -> Seq Scan on prt6_p1 prt6_5
+ -> Hash Join
+ Hash Cond: (prt5_3.a = sq1.aid)
+ -> Seq Scan on prt5_p2 prt5_3
+ -> Hash
+ -> Subquery Scan on sq1
+ Filter: (sq1.alabel ~~ '%abc%'::text)
+ -> Limit
+ -> Append
+ -> Seq Scan on prt6_p0 prt6_7
+ -> Seq Scan on prt6_p1 prt6_8
+(31 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT *
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+ QUERY PLAN
+------------------------------------------------------------
+ Append
+ -> Hash Join
+ Hash Cond: (prt5_1.a = t5_1.aid)
+ -> Hash Join
+ Hash Cond: (prt5_1.b = t5_2.bid)
+ -> Seq Scan on prt5_p0 prt5_1
+ -> Hash
+ -> Seq Scan on t5_2
+ Filter: (blabel ~~ '%cd%'::text)
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%ab%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_2.a = t5_1.aid)
+ -> Hash Join
+ Hash Cond: (prt5_2.b = t5_2.bid)
+ -> Seq Scan on prt5_p1 prt5_2
+ -> Hash
+ -> Seq Scan on t5_2
+ Filter: (blabel ~~ '%cd%'::text)
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%ab%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_3.a = t5_1.aid)
+ -> Hash Join
+ Hash Cond: (prt5_3.b = t5_2.bid)
+ -> Seq Scan on prt5_p2 prt5_3
+ -> Hash
+ -> Seq Scan on t5_2
+ Filter: (blabel ~~ '%cd%'::text)
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%ab%'::text)
+(34 rows)
+
+-- unable to extract non-partitioned right relation
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 RIGHT JOIN t5_1 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+-----------------------------------------------
+ Hash Right Join
+ Hash Cond: (prt5.a = t5_1.aid)
+ Join Filter: (t5_1.alabel ~~ '%abc%'::text)
+ -> Append
+ -> Seq Scan on prt5_p0 prt5_1
+ -> Seq Scan on prt5_p1 prt5_2
+ -> Seq Scan on prt5_p2 prt5_3
+ -> Hash
+ -> Seq Scan on t5_1
+(9 rows)
+
+-- left side can be extracted, but no cost benefit
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 LEFT JOIN t5_1 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+-------------------------------------------------
+ Hash Left Join
+ Hash Cond: (prt5.a = t5_1.aid)
+ -> Append
+ -> Seq Scan on prt5_p0 prt5_1
+ -> Seq Scan on prt5_p1 prt5_2
+ -> Seq Scan on prt5_p2 prt5_3
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+(9 rows)
+
+-- validation of the results with/without asymmetric partition-wise join
+SELECT * INTO pg_temp.result01a
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+SELECT * INTO pg_temp.result02a
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+SET enable_partitionwise_join = off;
+SELECT * INTO pg_temp.result01b
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+SELECT * INTO pg_temp.result02b
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+SELECT * FROM pg_temp.result01a EXCEPT SELECT * FROM pg_temp.result01b;
+ hkey | a | b | aid | alabel
+------+---+---+-----+--------
+(0 rows)
+
+SELECT * FROM pg_temp.result02a EXCEPT SELECT * FROM pg_temp.result02b;
+ hkey | a | b | aid | alabel | bid | blabel
+------+---+---+-----+--------+-----+--------
+(0 rows)
+
+RESET max_parallel_workers_per_gather;
+SET enable_partitionwise_join = true;
+-- Parameterized path examples.
+CREATE TABLE prta (id integer, payload text) PARTITION BY HASH (id);
+CREATE TABLE prta1 PARTITION OF prta FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE prta2 PARTITION OF prta FOR VALUES WITH (modulus 2, remainder 1);
+CREATE UNIQUE INDEX ON prta1 (id);
+CREATE UNIQUE INDEX ON prta2 (id);
+INSERT INTO prta (id, payload)
+ (SELECT *, ('abc' || id)::text AS payload
+ FROM generate_series(1,1) AS id);
+CREATE TABLE prtb (id integer, payload text) PARTITION BY HASH (id);
+CREATE TABLE prtb1 PARTITION OF prtb FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE prtb2 PARTITION OF prtb FOR VALUES WITH (modulus 2, remainder 1);
+CREATE UNIQUE INDEX ON prtb1 (id);
+CREATE UNIQUE INDEX ON prtb2 (id);
+INSERT INTO prtb (id, payload)
+ (SELECT *, ('def' || id)::text AS payload
+ FROM generate_series(1,1000) AS id);
+CREATE TABLE e (id integer, payload text) PARTITION BY HASH (id);
+CREATE TABLE e1 PARTITION OF e FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE e2 PARTITION OF e FOR VALUES WITH (modulus 2, remainder 1);
+INSERT INTO e (id, payload)
+ (SELECT *, ('ghi' || id)::text AS payload
+ FROM generate_series(1,1000) AS id);
+CREATE UNIQUE INDEX ON e1 (id);
+CREATE UNIQUE INDEX ON e2 (id);
+ANALYZE prta,prtb,e;
+EXPLAIN (COSTS OFF)
+SELECT * FROM prta,prtb WHERE prta.id=prtb.id;
+ QUERY PLAN
+-----------------------------------------------------------
+ Append
+ -> Nested Loop
+ -> Seq Scan on prta1 prta_1
+ -> Index Scan using prtb1_id_idx on prtb1 prtb_1
+ Index Cond: (id = prta_1.id)
+ -> Nested Loop
+ -> Seq Scan on prta2 prta_2
+ -> Index Scan using prtb2_id_idx on prtb2 prtb_2
+ Index Cond: (id = prta_2.id)
+(9 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT * FROM prta,prtb,e WHERE prta.id=prtb.id AND prta.id=e.id;
+ QUERY PLAN
+-----------------------------------------------------------------
+ Append
+ -> Nested Loop
+ Join Filter: (prta_1.id = e_1.id)
+ -> Nested Loop
+ -> Seq Scan on prta1 prta_1
+ -> Index Scan using prtb1_id_idx on prtb1 prtb_1
+ Index Cond: (id = prta_1.id)
+ -> Index Scan using e1_id_idx on e1 e_1
+ Index Cond: (id = prtb_1.id)
+ -> Nested Loop
+ Join Filter: (prta_2.id = e_2.id)
+ -> Nested Loop
+ -> Seq Scan on prta2 prta_2
+ -> Index Scan using prtb2_id_idx on prtb2 prtb_2
+ Index Cond: (id = prta_2.id)
+ -> Index Scan using e2_id_idx on e2 e_2
+ Index Cond: (id = prtb_2.id)
+(17 rows)
+
-- semi join
EXPLAIN (COSTS OFF)
SELECT t1.* FROM prt1_adv t1 WHERE EXISTS (SELECT 1 FROM prt2_adv t2 WHERE t1.a = t2.b) AND t1.b = 0 ORDER BY t1.a;
diff --git a/src/test/regress/sql/partition_join.sql b/src/test/regress/sql/partition_join.sql
index d97b5b69ff..51b81ce3f8 100644
--- a/src/test/regress/sql/partition_join.sql
+++ b/src/test/regress/sql/partition_join.sql
@@ -536,6 +536,132 @@ EXPLAIN (COSTS OFF)
SELECT t1.a, t1.c, t2.b, t2.c FROM prt1_adv t1 INNER JOIN prt2_adv t2 ON (t1.a = t2.b) WHERE t1.b = 0 ORDER BY t1.a, t2.b;
SELECT t1.a, t1.c, t2.b, t2.c FROM prt1_adv t1 INNER JOIN prt2_adv t2 ON (t1.a = t2.b) WHERE t1.b = 0 ORDER BY t1.a, t2.b;
+--
+-- For asymmetric partition-wise join
+--
+CREATE TABLE prt5 (hkey int, a int, b int) PARTITION BY HASH(hkey);
+CREATE TABLE prt5_p0 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 0);
+CREATE TABLE prt5_p1 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 1);
+CREATE TABLE prt5_p2 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 2);
+CREATE TABLE prt6 (aid int, alabel text) PARTITION BY HASH(aid);
+CREATE TABLE prt6_p0 PARTITION OF prt6
+ FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE prt6_p1 PARTITION OF prt6
+ FOR VALUES WITH (modulus 2, remainder 1);
+CREATE TABLE t5_1 (aid int, alabel text);
+CREATE TABLE t5_2 (bid int, blabel text);
+
+INSERT INTO prt5 (SELECT x, (1000.0 * random())::int,
+ (1000.0 * random())::int
+ FROM generate_series(1,1000000) x);
+INSERT INTO t5_1 (SELECT x, md5(x::text) FROM generate_series(-200, 800) x);
+INSERT INTO t5_2 (SELECT x, md5(x::text) FROM generate_series(-200, 800) x);
+INSERT INTO prt6 (SELECT * FROM t5_1);
+
+VACUUM ANALYZE prt5,prt6,t5_1,t5_2;
+
+SET max_parallel_workers_per_gather = 0;
+
+-- Trivial asymmetric JOIN of partitioned table with a relation
+EXPLAIN (COSTS OFF)
+SELECT *
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+
+-- The same, but appended with UNION ALL
+EXPLAIN (COSTS OFF)
+SELECT * FROM (
+ (SELECT * FROM prt5_p0)
+ UNION ALL
+ (SELECT * FROM prt5_p1)
+ UNION ALL
+ (SELECT * FROM prt5_p2)
+ ) AS sq1
+JOIN t5_1 ON a = aid AND alabel like '%abc%';
+
+-- Don't allow asymmetric JOIN of two partitioned tables.
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 JOIN prt6 ON a = aid AND alabel like '%abc%';
+
+-- Check asymmetric JOIN with Subquery
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 JOIN (
+ SELECT * FROM prt6 LIMIT 1000
+) AS sq1 ON a = aid AND alabel like '%abc%';
+
+EXPLAIN (COSTS OFF)
+SELECT *
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+
+-- unable to extract non-partitioned right relation
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 RIGHT JOIN t5_1 ON a = aid AND alabel like '%abc%';
+-- left side can be extracted, but no cost benefit
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 LEFT JOIN t5_1 ON a = aid AND alabel like '%abc%';
+
+-- validation of the results with/without asymmetric partition-wise join
+SELECT * INTO pg_temp.result01a
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+
+SELECT * INTO pg_temp.result02a
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+
+SET enable_partitionwise_join = off;
+
+SELECT * INTO pg_temp.result01b
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+
+SELECT * INTO pg_temp.result02b
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+
+SELECT * FROM pg_temp.result01a EXCEPT SELECT * FROM pg_temp.result01b;
+SELECT * FROM pg_temp.result02a EXCEPT SELECT * FROM pg_temp.result02b;
+
+RESET max_parallel_workers_per_gather;
+SET enable_partitionwise_join = true;
+
+-- Parameterized path examples.
+CREATE TABLE prta (id integer, payload text) PARTITION BY HASH (id);
+CREATE TABLE prta1 PARTITION OF prta FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE prta2 PARTITION OF prta FOR VALUES WITH (modulus 2, remainder 1);
+CREATE UNIQUE INDEX ON prta1 (id);
+CREATE UNIQUE INDEX ON prta2 (id);
+INSERT INTO prta (id, payload)
+ (SELECT *, ('abc' || id)::text AS payload
+ FROM generate_series(1,1) AS id);
+
+CREATE TABLE prtb (id integer, payload text) PARTITION BY HASH (id);
+CREATE TABLE prtb1 PARTITION OF prtb FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE prtb2 PARTITION OF prtb FOR VALUES WITH (modulus 2, remainder 1);
+CREATE UNIQUE INDEX ON prtb1 (id);
+CREATE UNIQUE INDEX ON prtb2 (id);
+INSERT INTO prtb (id, payload)
+ (SELECT *, ('def' || id)::text AS payload
+ FROM generate_series(1,1000) AS id);
+
+CREATE TABLE e (id integer, payload text) PARTITION BY HASH (id);
+CREATE TABLE e1 PARTITION OF e FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE e2 PARTITION OF e FOR VALUES WITH (modulus 2, remainder 1);
+INSERT INTO e (id, payload)
+ (SELECT *, ('ghi' || id)::text AS payload
+ FROM generate_series(1,1000) AS id);
+CREATE UNIQUE INDEX ON e1 (id);
+CREATE UNIQUE INDEX ON e2 (id);
+
+ANALYZE prta,prtb,e;
+
+EXPLAIN (COSTS OFF)
+SELECT * FROM prta,prtb WHERE prta.id=prtb.id;
+
+EXPLAIN (COSTS OFF)
+SELECT * FROM prta,prtb,e WHERE prta.id=prtb.id AND prta.id=e.id;
+
-- semi join
EXPLAIN (COSTS OFF)
SELECT t1.* FROM prt1_adv t1 WHERE EXISTS (SELECT 1 FROM prt2_adv t2 WHERE t1.a = t2.b) AND t1.b = 0 ORDER BY t1.a;
--
2.31.1
Attachments:
[text/plain] v14-0001-Asymmetric-partitionwise-join.patch (33.0K, ../../[email protected]/2-v14-0001-Asymmetric-partitionwise-join.patch)
download | inline diff:
From 101614b504b0b17e201d2375c8af61cfc671e51d Mon Sep 17 00:00:00 2001
From: Andrey Lepikhov <[email protected]>
Date: Fri, 2 Apr 2021 11:02:20 +0500
Subject: [PATCH] Asymmetric partitionwise join.
Teach optimizer to consider partitionwise join of non-partitioned
table with each partition of partitioned table.
Disallow asymmetric machinery for joining of two partitioned (or appended)
relations because it could cause CPU and memory huge consumption
during reparameterization of NestLoop path.
---
src/backend/optimizer/path/joinpath.c | 9 +
src/backend/optimizer/path/joinrels.c | 184 +++++++++++
src/backend/optimizer/plan/setrefs.c | 13 +-
src/backend/optimizer/util/appendinfo.c | 37 ++-
src/backend/optimizer/util/pathnode.c | 9 +-
src/backend/optimizer/util/relnode.c | 19 +-
src/include/optimizer/paths.h | 7 +-
src/test/regress/expected/partition_join.out | 306 +++++++++++++++++++
src/test/regress/sql/partition_join.sql | 126 ++++++++
9 files changed, 682 insertions(+), 28 deletions(-)
diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c
index b67b517770..7a1a7b2e86 100644
--- a/src/backend/optimizer/path/joinpath.c
+++ b/src/backend/optimizer/path/joinpath.c
@@ -335,6 +335,15 @@ add_paths_to_joinrel(PlannerInfo *root,
if (set_join_pathlist_hook)
set_join_pathlist_hook(root, joinrel, outerrel, innerrel,
jointype, &extra);
+
+ /*
+ * 7. If outer relation is delivered from partition-tables, consider
+ * distributing inner relation to every partition-leaf prior to
+ * append these leafs.
+ */
+ try_asymmetric_partitionwise_join(root, joinrel,
+ outerrel, innerrel,
+ jointype, &extra);
}
/*
diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c
index 0dbe2ac726..e6bd2ea5fe 100644
--- a/src/backend/optimizer/path/joinrels.c
+++ b/src/backend/optimizer/path/joinrels.c
@@ -16,6 +16,7 @@
#include "miscadmin.h"
#include "optimizer/appendinfo.h"
+#include "optimizer/cost.h"
#include "optimizer/joininfo.h"
#include "optimizer/pathnode.h"
#include "optimizer/paths.h"
@@ -1551,6 +1552,189 @@ try_partitionwise_join(PlannerInfo *root, RelOptInfo *rel1, RelOptInfo *rel2,
}
}
+/*
+ * Build RelOptInfo on JOIN of each partition of the outer relation and the inner
+ * relation. Return List of such RelOptInfo's. Return NIL, if at least one of
+ * these JOINs are impossible to build.
+ */
+static List *
+extract_asymmetric_partitionwise_subjoin(PlannerInfo *root,
+ RelOptInfo *joinrel,
+ AppendPath *append_path,
+ RelOptInfo *inner_rel,
+ JoinType jointype,
+ JoinPathExtraData *extra)
+{
+ List *result = NIL;
+ ListCell *lc;
+
+ foreach (lc, append_path->subpaths)
+ {
+ Path *child_path = lfirst(lc);
+ RelOptInfo *child_rel = child_path->parent;
+ Relids child_join_relids;
+ Relids parent_relids;
+ RelOptInfo *child_join_rel;
+ SpecialJoinInfo *child_sjinfo;
+ List *child_restrictlist;
+
+ child_join_relids = bms_union(child_rel->relids, inner_rel->relids);
+ parent_relids = bms_union(append_path->path.parent->relids,
+ inner_rel->relids);
+
+ child_sjinfo = build_child_join_sjinfo(root, extra->sjinfo,
+ child_rel->relids,
+ inner_rel->relids);
+ child_restrictlist = (List *)
+ adjust_appendrel_attrs_multilevel(root, (Node *)extra->restrictlist,
+ child_join_relids, parent_relids);
+
+ child_join_rel = find_join_rel(root, child_join_relids);
+ if (!child_join_rel)
+ {
+ child_join_rel = build_child_join_rel(root,
+ child_rel,
+ inner_rel,
+ joinrel,
+ child_restrictlist,
+ child_sjinfo,
+ jointype);
+ if (!child_join_rel)
+ {
+ /*
+ * If can't build JOIN between inner relation and one of the outer
+ * partitions - return immediately.
+ */
+ return NIL;
+ }
+ }
+ else
+ {
+ /*
+ * TODO:
+ * Can't imagine situation when join relation already exists. But in
+ * the 'partition_join' regression test it happens.
+ * It may be an indicator of possible problems.
+ */
+ }
+
+ populate_joinrel_with_paths(root,
+ child_rel,
+ inner_rel,
+ child_join_rel,
+ child_sjinfo,
+ child_restrictlist);
+
+ /* Give up if asymmetric partition-wise join is not available */
+ if (child_join_rel->pathlist == NIL)
+ return NIL;
+
+ set_cheapest(child_join_rel);
+ result = lappend(result, child_join_rel);
+ }
+ return result;
+}
+
+static bool
+is_asymmetric_join_capable(PlannerInfo *root,
+ RelOptInfo *outer_rel,
+ RelOptInfo *inner_rel,
+ JoinType jointype)
+{
+ ListCell *lc;
+
+ if (jointype != JOIN_INNER && jointype != JOIN_LEFT)
+ return false;
+
+ if (IS_OTHER_REL(outer_rel) || IS_OTHER_REL(inner_rel))
+ return false;
+
+ /* Disallow recursive usage of asymmertic join machinery */
+ if (root->join_rel_level == NULL)
+ return false;
+
+ /*
+ * Don't allow asymmetric JOIN of two append subplans.
+ * In the case of a parameterized NL join, a reparameterization procedure will
+ * lead to large memory allocations and a CPU consumption:
+ * each reparameterize will induce subpath duplication, creating new
+ * ParamPathInfo instance and increasing of ppilist up to number of partitions
+ * in the inner. Also, if we have many partitions, each bitmapset
+ * variable will large and many leaks of such variable (caused by relid
+ * replacement) will highly increase memory consumption.
+ * So, we deny such paths for now.
+ */
+ foreach(lc, inner_rel->pathlist)
+ {
+ if (IsA(lfirst(lc), AppendPath))
+ return false;
+ }
+
+ return true;
+}
+
+void
+try_asymmetric_partitionwise_join(PlannerInfo *root,
+ RelOptInfo *joinrel,
+ RelOptInfo *outer_rel,
+ RelOptInfo *inner_rel,
+ JoinType jointype,
+ JoinPathExtraData *extra)
+{
+ ListCell *lc;
+
+ /*
+ * Try this kind of paths if we allow complex partitionwise joins and we know
+ * we can build this join safely.
+ */
+ if (!enable_partitionwise_join ||
+ !is_asymmetric_join_capable(root, outer_rel, inner_rel, jointype))
+ return;
+
+ foreach (lc, outer_rel->pathlist)
+ {
+ AppendPath *append_path = lfirst(lc);
+
+ /*
+ * MEMO: We assume this pathlist keeps at least one AppendPath that
+ * represents partitioned table-scan, symmetric or asymmetric
+ * partition-wise join. It is not correct right now, however, a hook
+ * on add_path() to give additional decision for path removal allows
+ * to retain this kind of AppendPath, regardless of its cost.
+ */
+ if (IsA(append_path, AppendPath))
+ {
+ List **join_rel_level_saved;
+ List *live_childrels = NIL;
+
+ join_rel_level_saved = root->join_rel_level;
+ PG_TRY();
+ {
+ /* temporary disables "dynamic programming" algorithm */
+ root->join_rel_level = NULL;
+
+ live_childrels =
+ extract_asymmetric_partitionwise_subjoin(root,
+ joinrel,
+ append_path,
+ inner_rel,
+ jointype,
+ extra);
+ }
+ PG_FINALLY();
+ {
+ root->join_rel_level = join_rel_level_saved;
+ }
+ PG_END_TRY();
+
+ if (live_childrels != NIL)
+ add_paths_to_append_rel(root, joinrel, live_childrels);
+
+ break;
+ }
+ }
+}
+
/*
* Construct the SpecialJoinInfo for a child-join by translating
* SpecialJoinInfo for the join between parents. left_relids and right_relids
diff --git a/src/backend/optimizer/plan/setrefs.c b/src/backend/optimizer/plan/setrefs.c
index 61ccfd300b..dbe568aa29 100644
--- a/src/backend/optimizer/plan/setrefs.c
+++ b/src/backend/optimizer/plan/setrefs.c
@@ -287,18 +287,23 @@ set_plan_references(PlannerInfo *root, Plan *plan)
foreach(lc, root->append_rel_list)
{
AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc);
+ AppendRelInfo *newappinfo;
+
+ /* flat copy is enough since all valuable fields are scalars */
+ newappinfo = (AppendRelInfo *) palloc(sizeof(AppendRelInfo));
+ memcpy(newappinfo, appinfo, sizeof(AppendRelInfo));
/* adjust RT indexes */
- appinfo->parent_relid += rtoffset;
- appinfo->child_relid += rtoffset;
+ newappinfo->parent_relid += rtoffset;
+ newappinfo->child_relid += rtoffset;
/*
* Rather than adjust the translated_vars entries, just drop 'em.
* Neither the executor nor EXPLAIN currently need that data.
*/
- appinfo->translated_vars = NIL;
+ newappinfo->translated_vars = NIL;
- glob->appendRelations = lappend(glob->appendRelations, appinfo);
+ glob->appendRelations = lappend(glob->appendRelations, newappinfo);
}
/* Now fix the Plan tree */
diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c
index af46f581ac..ec8dee99b0 100644
--- a/src/backend/optimizer/util/appendinfo.c
+++ b/src/backend/optimizer/util/appendinfo.c
@@ -202,7 +202,9 @@ adjust_appendrel_attrs(PlannerInfo *root, Node *node, int nappinfos,
context.appinfos = appinfos;
/* If there's nothing to adjust, don't call this function. */
- Assert(nappinfos >= 1 && appinfos != NULL);
+ /* If there's nothing to adjust, just return a duplication */
+ if (nappinfos == 0)
+ return copyObject(node);
/* Should never be translating a Query tree. */
Assert(node == NULL || !IsA(node, Query));
@@ -489,12 +491,10 @@ adjust_appendrel_attrs_multilevel(PlannerInfo *root, Node *node,
Relids child_relids,
Relids top_parent_relids)
{
- AppendRelInfo **appinfos;
- Bitmapset *parent_relids = NULL;
- int nappinfos;
- int cnt;
-
- Assert(bms_num_members(child_relids) == bms_num_members(top_parent_relids));
+ AppendRelInfo **appinfos;
+ Relids parent_relids = NULL;
+ int nappinfos;
+ int cnt;
appinfos = find_appinfos_by_relids(root, child_relids, &nappinfos);
@@ -506,8 +506,13 @@ adjust_appendrel_attrs_multilevel(PlannerInfo *root, Node *node,
parent_relids = bms_add_member(parent_relids, appinfo->parent_relid);
}
- /* Recurse if immediate parent is not the top parent. */
- if (!bms_equal(parent_relids, top_parent_relids))
+ /*
+ * Recurse if immediate parent is not the top parent. Keep in mind that in a
+ * case of asymmetric JOIN top_parent_relids can contain relids which aren't
+ * part of an append node.
+ */
+ if (!bms_equal(parent_relids, top_parent_relids) &&
+ !bms_is_subset(parent_relids, top_parent_relids))
node = adjust_appendrel_attrs_multilevel(root, node, parent_relids,
top_parent_relids);
@@ -515,6 +520,7 @@ adjust_appendrel_attrs_multilevel(PlannerInfo *root, Node *node,
node = adjust_appendrel_attrs(root, node, nappinfos, appinfos);
pfree(appinfos);
+ pfree(parent_relids);
return node;
}
@@ -565,6 +571,7 @@ adjust_child_relids_multilevel(PlannerInfo *root, Relids relids,
AppendRelInfo **appinfos;
int nappinfos;
Relids parent_relids = NULL;
+ Relids normal_relids = NULL;
Relids result;
Relids tmp_result = NULL;
int cnt;
@@ -579,12 +586,17 @@ adjust_child_relids_multilevel(PlannerInfo *root, Relids relids,
appinfos = find_appinfos_by_relids(root, child_relids, &nappinfos);
/* Construct relids set for the immediate parent of the given child. */
+ normal_relids = bms_copy(child_relids);
for (cnt = 0; cnt < nappinfos; cnt++)
{
AppendRelInfo *appinfo = appinfos[cnt];
parent_relids = bms_add_member(parent_relids, appinfo->parent_relid);
+ normal_relids = bms_del_member(normal_relids, appinfo->child_relid);
}
+ parent_relids = bms_union(parent_relids, normal_relids);
+ if (normal_relids)
+ bms_free(normal_relids);
/* Recurse if immediate parent is not the top parent. */
if (!bms_equal(parent_relids, top_parent_relids))
@@ -715,11 +727,11 @@ AppendRelInfo **
find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos)
{
AppendRelInfo **appinfos;
+ int nrooms = bms_num_members(relids);
int cnt = 0;
int i;
- *nappinfos = bms_num_members(relids);
- appinfos = (AppendRelInfo **) palloc(sizeof(AppendRelInfo *) * *nappinfos);
+ appinfos = (AppendRelInfo **) palloc(sizeof(AppendRelInfo *) * nrooms);
i = -1;
while ((i = bms_next_member(relids, i)) >= 0)
@@ -727,10 +739,11 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos)
AppendRelInfo *appinfo = root->append_rel_array[i];
if (!appinfo)
- elog(ERROR, "child rel %d not found in append_rel_array", i);
+ continue;
appinfos[cnt++] = appinfo;
}
+ *nappinfos = cnt;
return appinfos;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index b248b038e0..73cfb4748d 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -4209,7 +4209,14 @@ do { \
MemoryContextSwitchTo(oldcontext);
}
- bms_free(required_outer);
+
+ /*
+ * If adjust_child_relids_multilevel don't do replacements it returns
+ * the original set, not a copy. It is possible in the case of asymmetric
+ * JOIN and child_rel->relids contains relids only of plane relations.
+ */
+ if (required_outer != old_ppi->ppi_req_outer)
+ bms_free(required_outer);
new_path->param_info = new_ppi;
diff --git a/src/backend/optimizer/util/relnode.c b/src/backend/optimizer/util/relnode.c
index e105a4d5f1..bb15fb2716 100644
--- a/src/backend/optimizer/util/relnode.c
+++ b/src/backend/optimizer/util/relnode.c
@@ -790,11 +790,8 @@ build_child_join_rel(PlannerInfo *root, RelOptInfo *outer_rel,
AppendRelInfo **appinfos;
int nappinfos;
- /* Only joins between "other" relations land here. */
- Assert(IS_OTHER_REL(outer_rel) && IS_OTHER_REL(inner_rel));
-
- /* The parent joinrel should have consider_partitionwise_join set. */
- Assert(parent_joinrel->consider_partitionwise_join);
+ /* Either of relations must be "other" relation at least. */
+ Assert(IS_OTHER_REL(outer_rel) || IS_OTHER_REL(inner_rel));
joinrel->reloptkind = RELOPT_OTHER_JOINREL;
joinrel->relids = bms_union(outer_rel->relids, inner_rel->relids);
@@ -851,8 +848,11 @@ build_child_join_rel(PlannerInfo *root, RelOptInfo *outer_rel,
joinrel->partexprs = NULL;
joinrel->nullable_partexprs = NULL;
- joinrel->top_parent_relids = bms_union(outer_rel->top_parent_relids,
- inner_rel->top_parent_relids);
+ joinrel->top_parent_relids =
+ bms_union(IS_OTHER_REL(outer_rel) ?
+ outer_rel->top_parent_relids : outer_rel->relids,
+ IS_OTHER_REL(inner_rel) ?
+ inner_rel->top_parent_relids : inner_rel->relids);
/* Compute information relevant to foreign relations. */
set_foreign_rel_properties(joinrel, outer_rel, inner_rel);
@@ -2033,9 +2033,8 @@ build_child_join_reltarget(PlannerInfo *root,
{
/* Build the targetlist */
childrel->reltarget->exprs = (List *)
- adjust_appendrel_attrs(root,
- (Node *) parentrel->reltarget->exprs,
- nappinfos, appinfos);
+ adjust_appendrel_attrs_multilevel(root, (Node *)parentrel->reltarget->exprs,
+ childrel->relids, parentrel->relids);
/* Set the cost and width fields */
childrel->reltarget->cost.startup = parentrel->reltarget->cost.startup;
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index f1d111063c..a0106dc539 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -109,7 +109,12 @@ extern bool have_join_order_restriction(PlannerInfo *root,
extern bool have_dangerous_phv(PlannerInfo *root,
Relids outer_relids, Relids inner_params);
extern void mark_dummy_rel(RelOptInfo *rel);
-
+extern void try_asymmetric_partitionwise_join(PlannerInfo *root,
+ RelOptInfo *joinrel,
+ RelOptInfo *outer_rel,
+ RelOptInfo *inner_rel,
+ JoinType jointype,
+ JoinPathExtraData *extra);
/*
* equivclass.c
* routines for managing EquivalenceClasses
diff --git a/src/test/regress/expected/partition_join.out b/src/test/regress/expected/partition_join.out
index 27f7525b3e..601f08662e 100644
--- a/src/test/regress/expected/partition_join.out
+++ b/src/test/regress/expected/partition_join.out
@@ -2320,6 +2320,312 @@ SELECT t1.a, t1.c, t2.b, t2.c FROM prt1_adv t1 INNER JOIN prt2_adv t2 ON (t1.a =
375 | 0375 | 375 | 0375
(8 rows)
+--
+-- For asymmetric partition-wise join
+--
+CREATE TABLE prt5 (hkey int, a int, b int) PARTITION BY HASH(hkey);
+CREATE TABLE prt5_p0 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 0);
+CREATE TABLE prt5_p1 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 1);
+CREATE TABLE prt5_p2 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 2);
+CREATE TABLE prt6 (aid int, alabel text) PARTITION BY HASH(aid);
+CREATE TABLE prt6_p0 PARTITION OF prt6
+ FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE prt6_p1 PARTITION OF prt6
+ FOR VALUES WITH (modulus 2, remainder 1);
+CREATE TABLE t5_1 (aid int, alabel text);
+CREATE TABLE t5_2 (bid int, blabel text);
+INSERT INTO prt5 (SELECT x, (1000.0 * random())::int,
+ (1000.0 * random())::int
+ FROM generate_series(1,1000000) x);
+INSERT INTO t5_1 (SELECT x, md5(x::text) FROM generate_series(-200, 800) x);
+INSERT INTO t5_2 (SELECT x, md5(x::text) FROM generate_series(-200, 800) x);
+INSERT INTO prt6 (SELECT * FROM t5_1);
+VACUUM ANALYZE prt5,prt6,t5_1,t5_2;
+SET max_parallel_workers_per_gather = 0;
+-- Trivial asymmetric JOIN of partitioned table with a relation
+EXPLAIN (COSTS OFF)
+SELECT *
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+-------------------------------------------------------
+ Append
+ -> Hash Join
+ Hash Cond: (prt5_1.a = t5_1.aid)
+ -> Seq Scan on prt5_p0 prt5_1
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_2.a = t5_1.aid)
+ -> Seq Scan on prt5_p1 prt5_2
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_3.a = t5_1.aid)
+ -> Seq Scan on prt5_p2 prt5_3
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+(19 rows)
+
+-- The same, but appended with UNION ALL
+EXPLAIN (COSTS OFF)
+SELECT * FROM (
+ (SELECT * FROM prt5_p0)
+ UNION ALL
+ (SELECT * FROM prt5_p1)
+ UNION ALL
+ (SELECT * FROM prt5_p2)
+ ) AS sq1
+JOIN t5_1 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+-------------------------------------------------------
+ Append
+ -> Hash Join
+ Hash Cond: (prt5_p0.a = t5_1.aid)
+ -> Seq Scan on prt5_p0
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_p1.a = t5_1.aid)
+ -> Seq Scan on prt5_p1
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_p2.a = t5_1.aid)
+ -> Seq Scan on prt5_p2
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+(19 rows)
+
+-- Don't allow asymmetric JOIN of two partitioned tables.
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 JOIN prt6 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+-------------------------------------------------------
+ Hash Join
+ Hash Cond: (prt5.a = prt6.aid)
+ -> Append
+ -> Seq Scan on prt5_p0 prt5_1
+ -> Seq Scan on prt5_p1 prt5_2
+ -> Seq Scan on prt5_p2 prt5_3
+ -> Hash
+ -> Append
+ -> Seq Scan on prt6_p0 prt6_1
+ Filter: (alabel ~~ '%abc%'::text)
+ -> Seq Scan on prt6_p1 prt6_2
+ Filter: (alabel ~~ '%abc%'::text)
+(12 rows)
+
+-- Check asymmetric JOIN with Subquery
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 JOIN (
+ SELECT * FROM prt6 LIMIT 1000
+) AS sq1 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+----------------------------------------------------------------
+ Append
+ -> Hash Join
+ Hash Cond: (prt5_1.a = sq1.aid)
+ -> Seq Scan on prt5_p0 prt5_1
+ -> Hash
+ -> Subquery Scan on sq1
+ Filter: (sq1.alabel ~~ '%abc%'::text)
+ -> Limit
+ -> Append
+ -> Seq Scan on prt6_p0 prt6_1
+ -> Seq Scan on prt6_p1 prt6_2
+ -> Hash Join
+ Hash Cond: (prt5_2.a = sq1.aid)
+ -> Seq Scan on prt5_p1 prt5_2
+ -> Hash
+ -> Subquery Scan on sq1
+ Filter: (sq1.alabel ~~ '%abc%'::text)
+ -> Limit
+ -> Append
+ -> Seq Scan on prt6_p0 prt6_4
+ -> Seq Scan on prt6_p1 prt6_5
+ -> Hash Join
+ Hash Cond: (prt5_3.a = sq1.aid)
+ -> Seq Scan on prt5_p2 prt5_3
+ -> Hash
+ -> Subquery Scan on sq1
+ Filter: (sq1.alabel ~~ '%abc%'::text)
+ -> Limit
+ -> Append
+ -> Seq Scan on prt6_p0 prt6_7
+ -> Seq Scan on prt6_p1 prt6_8
+(31 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT *
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+ QUERY PLAN
+------------------------------------------------------------
+ Append
+ -> Hash Join
+ Hash Cond: (prt5_1.a = t5_1.aid)
+ -> Hash Join
+ Hash Cond: (prt5_1.b = t5_2.bid)
+ -> Seq Scan on prt5_p0 prt5_1
+ -> Hash
+ -> Seq Scan on t5_2
+ Filter: (blabel ~~ '%cd%'::text)
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%ab%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_2.a = t5_1.aid)
+ -> Hash Join
+ Hash Cond: (prt5_2.b = t5_2.bid)
+ -> Seq Scan on prt5_p1 prt5_2
+ -> Hash
+ -> Seq Scan on t5_2
+ Filter: (blabel ~~ '%cd%'::text)
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%ab%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_3.a = t5_1.aid)
+ -> Hash Join
+ Hash Cond: (prt5_3.b = t5_2.bid)
+ -> Seq Scan on prt5_p2 prt5_3
+ -> Hash
+ -> Seq Scan on t5_2
+ Filter: (blabel ~~ '%cd%'::text)
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%ab%'::text)
+(34 rows)
+
+-- unable to extract non-partitioned right relation
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 RIGHT JOIN t5_1 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+-----------------------------------------------
+ Hash Right Join
+ Hash Cond: (prt5.a = t5_1.aid)
+ Join Filter: (t5_1.alabel ~~ '%abc%'::text)
+ -> Append
+ -> Seq Scan on prt5_p0 prt5_1
+ -> Seq Scan on prt5_p1 prt5_2
+ -> Seq Scan on prt5_p2 prt5_3
+ -> Hash
+ -> Seq Scan on t5_1
+(9 rows)
+
+-- left side can be extracted, but no cost benefit
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 LEFT JOIN t5_1 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+-------------------------------------------------
+ Hash Left Join
+ Hash Cond: (prt5.a = t5_1.aid)
+ -> Append
+ -> Seq Scan on prt5_p0 prt5_1
+ -> Seq Scan on prt5_p1 prt5_2
+ -> Seq Scan on prt5_p2 prt5_3
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+(9 rows)
+
+-- validation of the results with/without asymmetric partition-wise join
+SELECT * INTO pg_temp.result01a
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+SELECT * INTO pg_temp.result02a
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+SET enable_partitionwise_join = off;
+SELECT * INTO pg_temp.result01b
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+SELECT * INTO pg_temp.result02b
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+SELECT * FROM pg_temp.result01a EXCEPT SELECT * FROM pg_temp.result01b;
+ hkey | a | b | aid | alabel
+------+---+---+-----+--------
+(0 rows)
+
+SELECT * FROM pg_temp.result02a EXCEPT SELECT * FROM pg_temp.result02b;
+ hkey | a | b | aid | alabel | bid | blabel
+------+---+---+-----+--------+-----+--------
+(0 rows)
+
+RESET max_parallel_workers_per_gather;
+SET enable_partitionwise_join = true;
+-- Parameterized path examples.
+CREATE TABLE prta (id integer, payload text) PARTITION BY HASH (id);
+CREATE TABLE prta1 PARTITION OF prta FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE prta2 PARTITION OF prta FOR VALUES WITH (modulus 2, remainder 1);
+CREATE UNIQUE INDEX ON prta1 (id);
+CREATE UNIQUE INDEX ON prta2 (id);
+INSERT INTO prta (id, payload)
+ (SELECT *, ('abc' || id)::text AS payload
+ FROM generate_series(1,1) AS id);
+CREATE TABLE prtb (id integer, payload text) PARTITION BY HASH (id);
+CREATE TABLE prtb1 PARTITION OF prtb FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE prtb2 PARTITION OF prtb FOR VALUES WITH (modulus 2, remainder 1);
+CREATE UNIQUE INDEX ON prtb1 (id);
+CREATE UNIQUE INDEX ON prtb2 (id);
+INSERT INTO prtb (id, payload)
+ (SELECT *, ('def' || id)::text AS payload
+ FROM generate_series(1,1000) AS id);
+CREATE TABLE e (id integer, payload text) PARTITION BY HASH (id);
+CREATE TABLE e1 PARTITION OF e FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE e2 PARTITION OF e FOR VALUES WITH (modulus 2, remainder 1);
+INSERT INTO e (id, payload)
+ (SELECT *, ('ghi' || id)::text AS payload
+ FROM generate_series(1,1000) AS id);
+CREATE UNIQUE INDEX ON e1 (id);
+CREATE UNIQUE INDEX ON e2 (id);
+ANALYZE prta,prtb,e;
+EXPLAIN (COSTS OFF)
+SELECT * FROM prta,prtb WHERE prta.id=prtb.id;
+ QUERY PLAN
+-----------------------------------------------------------
+ Append
+ -> Nested Loop
+ -> Seq Scan on prta1 prta_1
+ -> Index Scan using prtb1_id_idx on prtb1 prtb_1
+ Index Cond: (id = prta_1.id)
+ -> Nested Loop
+ -> Seq Scan on prta2 prta_2
+ -> Index Scan using prtb2_id_idx on prtb2 prtb_2
+ Index Cond: (id = prta_2.id)
+(9 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT * FROM prta,prtb,e WHERE prta.id=prtb.id AND prta.id=e.id;
+ QUERY PLAN
+-----------------------------------------------------------------
+ Append
+ -> Nested Loop
+ Join Filter: (prta_1.id = e_1.id)
+ -> Nested Loop
+ -> Seq Scan on prta1 prta_1
+ -> Index Scan using prtb1_id_idx on prtb1 prtb_1
+ Index Cond: (id = prta_1.id)
+ -> Index Scan using e1_id_idx on e1 e_1
+ Index Cond: (id = prtb_1.id)
+ -> Nested Loop
+ Join Filter: (prta_2.id = e_2.id)
+ -> Nested Loop
+ -> Seq Scan on prta2 prta_2
+ -> Index Scan using prtb2_id_idx on prtb2 prtb_2
+ Index Cond: (id = prta_2.id)
+ -> Index Scan using e2_id_idx on e2 e_2
+ Index Cond: (id = prtb_2.id)
+(17 rows)
+
-- semi join
EXPLAIN (COSTS OFF)
SELECT t1.* FROM prt1_adv t1 WHERE EXISTS (SELECT 1 FROM prt2_adv t2 WHERE t1.a = t2.b) AND t1.b = 0 ORDER BY t1.a;
diff --git a/src/test/regress/sql/partition_join.sql b/src/test/regress/sql/partition_join.sql
index d97b5b69ff..51b81ce3f8 100644
--- a/src/test/regress/sql/partition_join.sql
+++ b/src/test/regress/sql/partition_join.sql
@@ -536,6 +536,132 @@ EXPLAIN (COSTS OFF)
SELECT t1.a, t1.c, t2.b, t2.c FROM prt1_adv t1 INNER JOIN prt2_adv t2 ON (t1.a = t2.b) WHERE t1.b = 0 ORDER BY t1.a, t2.b;
SELECT t1.a, t1.c, t2.b, t2.c FROM prt1_adv t1 INNER JOIN prt2_adv t2 ON (t1.a = t2.b) WHERE t1.b = 0 ORDER BY t1.a, t2.b;
+--
+-- For asymmetric partition-wise join
+--
+CREATE TABLE prt5 (hkey int, a int, b int) PARTITION BY HASH(hkey);
+CREATE TABLE prt5_p0 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 0);
+CREATE TABLE prt5_p1 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 1);
+CREATE TABLE prt5_p2 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 2);
+CREATE TABLE prt6 (aid int, alabel text) PARTITION BY HASH(aid);
+CREATE TABLE prt6_p0 PARTITION OF prt6
+ FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE prt6_p1 PARTITION OF prt6
+ FOR VALUES WITH (modulus 2, remainder 1);
+CREATE TABLE t5_1 (aid int, alabel text);
+CREATE TABLE t5_2 (bid int, blabel text);
+
+INSERT INTO prt5 (SELECT x, (1000.0 * random())::int,
+ (1000.0 * random())::int
+ FROM generate_series(1,1000000) x);
+INSERT INTO t5_1 (SELECT x, md5(x::text) FROM generate_series(-200, 800) x);
+INSERT INTO t5_2 (SELECT x, md5(x::text) FROM generate_series(-200, 800) x);
+INSERT INTO prt6 (SELECT * FROM t5_1);
+
+VACUUM ANALYZE prt5,prt6,t5_1,t5_2;
+
+SET max_parallel_workers_per_gather = 0;
+
+-- Trivial asymmetric JOIN of partitioned table with a relation
+EXPLAIN (COSTS OFF)
+SELECT *
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+
+-- The same, but appended with UNION ALL
+EXPLAIN (COSTS OFF)
+SELECT * FROM (
+ (SELECT * FROM prt5_p0)
+ UNION ALL
+ (SELECT * FROM prt5_p1)
+ UNION ALL
+ (SELECT * FROM prt5_p2)
+ ) AS sq1
+JOIN t5_1 ON a = aid AND alabel like '%abc%';
+
+-- Don't allow asymmetric JOIN of two partitioned tables.
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 JOIN prt6 ON a = aid AND alabel like '%abc%';
+
+-- Check asymmetric JOIN with Subquery
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 JOIN (
+ SELECT * FROM prt6 LIMIT 1000
+) AS sq1 ON a = aid AND alabel like '%abc%';
+
+EXPLAIN (COSTS OFF)
+SELECT *
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+
+-- unable to extract non-partitioned right relation
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 RIGHT JOIN t5_1 ON a = aid AND alabel like '%abc%';
+-- left side can be extracted, but no cost benefit
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 LEFT JOIN t5_1 ON a = aid AND alabel like '%abc%';
+
+-- validation of the results with/without asymmetric partition-wise join
+SELECT * INTO pg_temp.result01a
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+
+SELECT * INTO pg_temp.result02a
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+
+SET enable_partitionwise_join = off;
+
+SELECT * INTO pg_temp.result01b
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+
+SELECT * INTO pg_temp.result02b
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+
+SELECT * FROM pg_temp.result01a EXCEPT SELECT * FROM pg_temp.result01b;
+SELECT * FROM pg_temp.result02a EXCEPT SELECT * FROM pg_temp.result02b;
+
+RESET max_parallel_workers_per_gather;
+SET enable_partitionwise_join = true;
+
+-- Parameterized path examples.
+CREATE TABLE prta (id integer, payload text) PARTITION BY HASH (id);
+CREATE TABLE prta1 PARTITION OF prta FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE prta2 PARTITION OF prta FOR VALUES WITH (modulus 2, remainder 1);
+CREATE UNIQUE INDEX ON prta1 (id);
+CREATE UNIQUE INDEX ON prta2 (id);
+INSERT INTO prta (id, payload)
+ (SELECT *, ('abc' || id)::text AS payload
+ FROM generate_series(1,1) AS id);
+
+CREATE TABLE prtb (id integer, payload text) PARTITION BY HASH (id);
+CREATE TABLE prtb1 PARTITION OF prtb FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE prtb2 PARTITION OF prtb FOR VALUES WITH (modulus 2, remainder 1);
+CREATE UNIQUE INDEX ON prtb1 (id);
+CREATE UNIQUE INDEX ON prtb2 (id);
+INSERT INTO prtb (id, payload)
+ (SELECT *, ('def' || id)::text AS payload
+ FROM generate_series(1,1000) AS id);
+
+CREATE TABLE e (id integer, payload text) PARTITION BY HASH (id);
+CREATE TABLE e1 PARTITION OF e FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE e2 PARTITION OF e FOR VALUES WITH (modulus 2, remainder 1);
+INSERT INTO e (id, payload)
+ (SELECT *, ('ghi' || id)::text AS payload
+ FROM generate_series(1,1000) AS id);
+CREATE UNIQUE INDEX ON e1 (id);
+CREATE UNIQUE INDEX ON e2 (id);
+
+ANALYZE prta,prtb,e;
+
+EXPLAIN (COSTS OFF)
+SELECT * FROM prta,prtb WHERE prta.id=prtb.id;
+
+EXPLAIN (COSTS OFF)
+SELECT * FROM prta,prtb,e WHERE prta.id=prtb.id AND prta.id=e.id;
+
-- semi join
EXPLAIN (COSTS OFF)
SELECT t1.* FROM prt1_adv t1 WHERE EXISTS (SELECT 1 FROM prt2_adv t2 WHERE t1.a = t2.b) AND t1.b = 0 ORDER BY t1.a;
--
2.31.1
^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: Asymmetric partition-wise JOIN
2019-08-12 06:03 Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-08-22 16:05 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-08-23 22:01 ` Re: Asymmetric partition-wise JOIN Thomas Munro <[email protected]>
2019-08-24 08:33 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-12-01 03:24 ` Re: Asymmetric partition-wise JOIN Michael Paquier <[email protected]>
2019-12-27 07:34 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2020-07-01 09:10 ` Re: Asymmetric partition-wise JOIN Daniel Gustafsson <[email protected]>
2020-08-21 06:02 ` Re: Asymmetric partition-wise JOIN Andrey V. Lepikhov <[email protected]>
2020-11-09 10:53 ` Re: Asymmetric partition-wise JOIN Anastasia Lubennikova <[email protected]>
2020-11-30 14:43 ` Re: Asymmetric partition-wise JOIN Anastasia Lubennikova <[email protected]>
2021-04-30 03:10 ` Re: Asymmetric partition-wise JOIN Andrey V. Lepikhov <[email protected]>
2021-05-27 04:27 ` Re: Asymmetric partition-wise JOIN Andrey Lepikhov <[email protected]>
@ 2021-06-18 12:02 ` Alexander Pyhalov <[email protected]>
2021-07-05 09:57 ` Re: Asymmetric partition-wise JOIN Andrey Lepikhov <[email protected]>
0 siblings, 1 reply; 28+ messages in thread
From: Alexander Pyhalov @ 2021-06-18 12:02 UTC (permalink / raw)
To: Andrey Lepikhov <[email protected]>; +Cc: Anastasia Lubennikova <[email protected]>; Daniel Gustafsson <[email protected]>; Kohei KaiGai <[email protected]>; Michael Paquier <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers
Andrey Lepikhov писал 2021-05-27 07:27:
> Next version of the patch.
> For searching any problems I forced this patch during 'make check'
> tests. Some bugs were found and fixed.
Hi.
I've tested this patch and haven't found issues, but I have some
comments.
src/backend/optimizer/path/joinrels.c:
1554
1555 /*
1556 * Build RelOptInfo on JOIN of each partition of the outer relation
and the inner
1557 * relation. Return List of such RelOptInfo's. Return NIL, if at
least one of
1558 * these JOINs are impossible to build.
1559 */
1560 static List *
1561 extract_asymmetric_partitionwise_subjoin(PlannerInfo *root,
1562
RelOptInfo *joinrel,
1563
AppendPath *append_path,
1564
RelOptInfo *inner_rel,
1565
JoinType jointype,
1566
JoinPathExtraData *extra)
1567 {
1568 List *result = NIL;
1569 ListCell *lc;
1570
1571 foreach (lc, append_path->subpaths)
1572 {
1573 Path *child_path = lfirst(lc);
1574 RelOptInfo *child_rel =
child_path->parent;
1575 Relids child_join_relids;
1576 Relids parent_relids;
1577 RelOptInfo *child_join_rel;
1578 SpecialJoinInfo *child_sjinfo;
1579 List *child_restrictlist;
Variable names - child_join_rel and child_join_relids seem to be
inconsistent with rest of the file (I see child_joinrelids in
try_partitionwise_join() and child_joinrel in try_partitionwise_join()
and get_matching_part_pairs()).
1595 child_join_rel = build_child_join_rel(root,
1596
child_rel,
1597
inner_rel,
1598
joinrel,
1599
child_restrictlist,
1600
child_sjinfo,
1601
jointype);
1602 if (!child_join_rel)
1603 {
1604 /*
1605 * If can't build JOIN between
inner relation and one of the outer
1606 * partitions - return immediately.
1607 */
1608 return NIL;
1609 }
When build_child_join_rel() can return NULL?
If I read code correctly, joinrel is created in the begining of
build_child_join_rel() with makeNode(), makeNode() wraps newNode() and
newNode() uses MemoryContextAllocZero()/MemoryContextAllocZeroAligned(),
which would error() on alloc() failure.
1637
1638 static bool
1639 is_asymmetric_join_capable(PlannerInfo *root,
1640 RelOptInfo
*outer_rel,
1641 RelOptInfo
*inner_rel,
1642 JoinType
jointype)
1643 {
Function misses a comment.
1656 /*
1657 * Don't allow asymmetric JOIN of two append subplans.
1658 * In the case of a parameterized NL join, a
reparameterization procedure will
1659 * lead to large memory allocations and a CPU consumption:
1660 * each reparameterize will induce subpath duplication,
creating new
1661 * ParamPathInfo instance and increasing of ppilist up to
number of partitions
1662 * in the inner. Also, if we have many partitions, each
bitmapset
1663 * variable will large and many leaks of such variable
(caused by relid
1664 * replacement) will highly increase memory consumption.
1665 * So, we deny such paths for now.
1666 */
Missing word:
each bitmapset variable will large => each bitmapset variable will be
large
1694 foreach (lc, outer_rel->pathlist)
1695 {
1696 AppendPath *append_path = lfirst(lc);
1697
1698 /*
1699 * MEMO: We assume this pathlist keeps at least one
AppendPath that
1700 * represents partitioned table-scan, symmetric or
asymmetric
1701 * partition-wise join. It is not correct right
now, however, a hook
1702 * on add_path() to give additional decision for
path removal allows
1703 * to retain this kind of AppendPath, regardless of
its cost.
1704 */
1705 if (IsA(append_path, AppendPath))
What hook do you refer to?
src/backend/optimizer/plan/setrefs.c:
282 /*
283 * Adjust RT indexes of AppendRelInfos and add to final
appendrels list.
284 * We assume the AppendRelInfos were built during planning
and don't need
285 * to be copied.
286 */
287 foreach(lc, root->append_rel_list)
288 {
289 AppendRelInfo *appinfo = lfirst_node(AppendRelInfo,
lc);
290 AppendRelInfo *newappinfo;
291
292 /* flat copy is enough since all valuable fields are
scalars */
293 newappinfo = (AppendRelInfo *)
palloc(sizeof(AppendRelInfo));
294 memcpy(newappinfo, appinfo, sizeof(AppendRelInfo));
You've changed function to copy appinfo, so now comment is incorrect.
src/backend/optimizer/util/appendinfo.c:
588 /* Construct relids set for the immediate parent of the
given child. */
589 normal_relids = bms_copy(child_relids);
590 for (cnt = 0; cnt < nappinfos; cnt++)
591 {
592 AppendRelInfo *appinfo = appinfos[cnt];
593
594 parent_relids = bms_add_member(parent_relids,
appinfo->parent_relid);
595 normal_relids = bms_del_member(normal_relids,
appinfo->child_relid);
596 }
597 parent_relids = bms_union(parent_relids, normal_relids);
Do I understand correctly that now parent_relids also contains relids of
relations from 'global' inner relation, which we join to childs?
--
Best regards,
Alexander Pyhalov,
Postgres Professional
^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: Asymmetric partition-wise JOIN
2019-08-12 06:03 Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-08-22 16:05 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-08-23 22:01 ` Re: Asymmetric partition-wise JOIN Thomas Munro <[email protected]>
2019-08-24 08:33 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-12-01 03:24 ` Re: Asymmetric partition-wise JOIN Michael Paquier <[email protected]>
2019-12-27 07:34 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2020-07-01 09:10 ` Re: Asymmetric partition-wise JOIN Daniel Gustafsson <[email protected]>
2020-08-21 06:02 ` Re: Asymmetric partition-wise JOIN Andrey V. Lepikhov <[email protected]>
2020-11-09 10:53 ` Re: Asymmetric partition-wise JOIN Anastasia Lubennikova <[email protected]>
2020-11-30 14:43 ` Re: Asymmetric partition-wise JOIN Anastasia Lubennikova <[email protected]>
2021-04-30 03:10 ` Re: Asymmetric partition-wise JOIN Andrey V. Lepikhov <[email protected]>
2021-05-27 04:27 ` Re: Asymmetric partition-wise JOIN Andrey Lepikhov <[email protected]>
2021-06-18 12:02 ` Re: Asymmetric partition-wise JOIN Alexander Pyhalov <[email protected]>
@ 2021-07-05 09:57 ` Andrey Lepikhov <[email protected]>
2021-07-05 20:15 ` Re: Asymmetric partition-wise JOIN Zhihong Yu <[email protected]>
0 siblings, 1 reply; 28+ messages in thread
From: Andrey Lepikhov @ 2021-07-05 09:57 UTC (permalink / raw)
To: Alexander Pyhalov <[email protected]>; +Cc: Anastasia Lubennikova <[email protected]>; Daniel Gustafsson <[email protected]>; Kohei KaiGai <[email protected]>; Michael Paquier <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers
On 18/6/21 15:02, Alexander Pyhalov wrote:
> Andrey Lepikhov писал 2021-05-27 07:27:
>> Next version of the patch.
>> For searching any problems I forced this patch during 'make check'
>> tests. Some bugs were found and fixed.
>
> Hi.
> I've tested this patch and haven't found issues, but I have some comments.
Thank you for review!
> Variable names - child_join_rel and child_join_relids seem to be
> inconsistent with rest of the file
fixed
> When build_child_join_rel() can return NULL?
Fixed
> Missing word:
> each bitmapset variable will large => each bitmapset variable will be large
Fixed
> What hook do you refer to?
Removed> You've changed function to copy appinfo, so now comment is
incorrect.
Thanks, fixed> Do I understand correctly that now parent_relids also
contains relids of
> relations from 'global' inner relation, which we join to childs?
Yes
--
regards,
Andrey Lepikhov
Postgres Professional
From d653b4f65486e2cefafda61811390c4095eae371 Mon Sep 17 00:00:00 2001
From: Andrey Lepikhov <[email protected]>
Date: Fri, 2 Apr 2021 11:02:20 +0500
Subject: [PATCH] Asymmetric partitionwise join.
Teach optimizer to consider partitionwise join of non-partitioned
table with each partition of partitioned table.
Disallow asymmetric machinery for joining of two partitioned (or appended)
relations because it could cause CPU and memory huge consumption
during reparameterization of NestLoop path.
---
src/backend/optimizer/path/joinpath.c | 9 +
src/backend/optimizer/path/joinrels.c | 172 +++++++++++
src/backend/optimizer/plan/setrefs.c | 17 +-
src/backend/optimizer/util/appendinfo.c | 37 ++-
src/backend/optimizer/util/pathnode.c | 9 +-
src/backend/optimizer/util/relnode.c | 19 +-
src/include/optimizer/paths.h | 7 +-
src/test/regress/expected/partition_join.out | 306 +++++++++++++++++++
src/test/regress/sql/partition_join.sql | 126 ++++++++
9 files changed, 672 insertions(+), 30 deletions(-)
diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c
index b67b517770..7a1a7b2e86 100644
--- a/src/backend/optimizer/path/joinpath.c
+++ b/src/backend/optimizer/path/joinpath.c
@@ -335,6 +335,15 @@ add_paths_to_joinrel(PlannerInfo *root,
if (set_join_pathlist_hook)
set_join_pathlist_hook(root, joinrel, outerrel, innerrel,
jointype, &extra);
+
+ /*
+ * 7. If outer relation is delivered from partition-tables, consider
+ * distributing inner relation to every partition-leaf prior to
+ * append these leafs.
+ */
+ try_asymmetric_partitionwise_join(root, joinrel,
+ outerrel, innerrel,
+ jointype, &extra);
}
/*
diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c
index 0dbe2ac726..2839b7acc3 100644
--- a/src/backend/optimizer/path/joinrels.c
+++ b/src/backend/optimizer/path/joinrels.c
@@ -16,6 +16,7 @@
#include "miscadmin.h"
#include "optimizer/appendinfo.h"
+#include "optimizer/cost.h"
#include "optimizer/joininfo.h"
#include "optimizer/pathnode.h"
#include "optimizer/paths.h"
@@ -1551,6 +1552,177 @@ try_partitionwise_join(PlannerInfo *root, RelOptInfo *rel1, RelOptInfo *rel2,
}
}
+/*
+ * Build RelOptInfo on JOIN of each partition of the outer relation and the inner
+ * relation. Return List of such RelOptInfo's. Return NIL, if at least one of
+ * these JOINs are impossible to build.
+ */
+static List *
+extract_asymmetric_partitionwise_subjoin(PlannerInfo *root,
+ RelOptInfo *joinrel,
+ AppendPath *append_path,
+ RelOptInfo *inner_rel,
+ JoinType jointype,
+ JoinPathExtraData *extra)
+{
+ List *result = NIL;
+ ListCell *lc;
+
+ foreach (lc, append_path->subpaths)
+ {
+ Path *child_path = lfirst(lc);
+ RelOptInfo *child_rel = child_path->parent;
+ Relids child_joinrelids;
+ Relids parent_relids;
+ RelOptInfo *child_joinrel;
+ SpecialJoinInfo *child_sjinfo;
+ List *child_restrictlist;
+
+ child_joinrelids = bms_union(child_rel->relids, inner_rel->relids);
+ parent_relids = bms_union(append_path->path.parent->relids,
+ inner_rel->relids);
+
+ child_sjinfo = build_child_join_sjinfo(root, extra->sjinfo,
+ child_rel->relids,
+ inner_rel->relids);
+ child_restrictlist = (List *)
+ adjust_appendrel_attrs_multilevel(root, (Node *)extra->restrictlist,
+ child_joinrelids, parent_relids);
+
+ child_joinrel = find_join_rel(root, child_joinrelids);
+ if (!child_joinrel)
+ child_joinrel = build_child_join_rel(root,
+ child_rel,
+ inner_rel,
+ joinrel,
+ child_restrictlist,
+ child_sjinfo,
+ jointype);
+ else
+ {
+ /*
+ * TODO:
+ * Can't imagine situation when join relation already exists. But in
+ * the 'partition_join' regression test it happens.
+ * It may be an indicator of possible problems.
+ */
+ }
+
+ populate_joinrel_with_paths(root,
+ child_rel,
+ inner_rel,
+ child_joinrel,
+ child_sjinfo,
+ child_restrictlist);
+
+ /* Give up if asymmetric partition-wise join is not available */
+ if (child_joinrel->pathlist == NIL)
+ return NIL;
+
+ set_cheapest(child_joinrel);
+ result = lappend(result, child_joinrel);
+ }
+ return result;
+}
+
+static bool
+is_asymmetric_join_capable(PlannerInfo *root,
+ RelOptInfo *outer_rel,
+ RelOptInfo *inner_rel,
+ JoinType jointype)
+{
+ ListCell *lc;
+
+ if (jointype != JOIN_INNER && jointype != JOIN_LEFT)
+ return false;
+
+ if (IS_OTHER_REL(outer_rel) || IS_OTHER_REL(inner_rel))
+ return false;
+
+ /* Disallow recursive usage of asymmertic join machinery */
+ if (root->join_rel_level == NULL)
+ return false;
+
+ /*
+ * Don't allow asymmetric JOIN of two append subplans.
+ * In the case of a parameterized NL join, a reparameterization procedure
+ * will lead to large memory allocations and a CPU consumption:
+ * each reparameterization will induce subpath duplication, creating new
+ * ParamPathInfo instance and increasing of ppilist up to number of
+ * partitions in the inner. Also, if we have many partitions, each bitmapset
+ * variable will be large and many leaks of such variable (caused by relid
+ * replacement) will highly increase memory consumption.
+ * So, we deny such paths for now.
+ */
+ foreach(lc, inner_rel->pathlist)
+ {
+ if (IsA(lfirst(lc), AppendPath))
+ return false;
+ }
+
+ return true;
+}
+
+void
+try_asymmetric_partitionwise_join(PlannerInfo *root,
+ RelOptInfo *joinrel,
+ RelOptInfo *outer_rel,
+ RelOptInfo *inner_rel,
+ JoinType jointype,
+ JoinPathExtraData *extra)
+{
+ ListCell *lc;
+
+ /*
+ * Try this kind of paths if we allow complex partitionwise joins and we know
+ * we can build this join safely.
+ */
+ if (!enable_partitionwise_join ||
+ !is_asymmetric_join_capable(root, outer_rel, inner_rel, jointype))
+ return;
+
+ foreach (lc, outer_rel->pathlist)
+ {
+ AppendPath *append_path = lfirst(lc);
+
+ /*
+ * We assume this pathlist keeps at least one AppendPath that
+ * represents partitioned table-scan, symmetric or asymmetric
+ * partition-wise join.
+ */
+ if (IsA(append_path, AppendPath))
+ {
+ List **join_rel_level_saved;
+ List *live_childrels = NIL;
+
+ join_rel_level_saved = root->join_rel_level;
+ PG_TRY();
+ {
+ /* temporary disables "dynamic programming" algorithm */
+ root->join_rel_level = NULL;
+
+ live_childrels =
+ extract_asymmetric_partitionwise_subjoin(root,
+ joinrel,
+ append_path,
+ inner_rel,
+ jointype,
+ extra);
+ }
+ PG_FINALLY();
+ {
+ root->join_rel_level = join_rel_level_saved;
+ }
+ PG_END_TRY();
+
+ if (live_childrels != NIL)
+ add_paths_to_append_rel(root, joinrel, live_childrels);
+
+ break;
+ }
+ }
+}
+
/*
* Construct the SpecialJoinInfo for a child-join by translating
* SpecialJoinInfo for the join between parents. left_relids and right_relids
diff --git a/src/backend/optimizer/plan/setrefs.c b/src/backend/optimizer/plan/setrefs.c
index 61ccfd300b..834843ba33 100644
--- a/src/backend/optimizer/plan/setrefs.c
+++ b/src/backend/optimizer/plan/setrefs.c
@@ -281,24 +281,29 @@ set_plan_references(PlannerInfo *root, Plan *plan)
/*
* Adjust RT indexes of AppendRelInfos and add to final appendrels list.
- * We assume the AppendRelInfos were built during planning and don't need
- * to be copied.
+ * The AppendRelInfos are copied, because as a part of a subplan its could
+ * be visited many times in the case of asymmetric join.
*/
foreach(lc, root->append_rel_list)
{
AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc);
+ AppendRelInfo *newappinfo;
+
+ /* flat copy is enough since all valuable fields are scalars */
+ newappinfo = (AppendRelInfo *) palloc(sizeof(AppendRelInfo));
+ memcpy(newappinfo, appinfo, sizeof(AppendRelInfo));
/* adjust RT indexes */
- appinfo->parent_relid += rtoffset;
- appinfo->child_relid += rtoffset;
+ newappinfo->parent_relid += rtoffset;
+ newappinfo->child_relid += rtoffset;
/*
* Rather than adjust the translated_vars entries, just drop 'em.
* Neither the executor nor EXPLAIN currently need that data.
*/
- appinfo->translated_vars = NIL;
+ newappinfo->translated_vars = NIL;
- glob->appendRelations = lappend(glob->appendRelations, appinfo);
+ glob->appendRelations = lappend(glob->appendRelations, newappinfo);
}
/* Now fix the Plan tree */
diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c
index af46f581ac..ec8dee99b0 100644
--- a/src/backend/optimizer/util/appendinfo.c
+++ b/src/backend/optimizer/util/appendinfo.c
@@ -202,7 +202,9 @@ adjust_appendrel_attrs(PlannerInfo *root, Node *node, int nappinfos,
context.appinfos = appinfos;
/* If there's nothing to adjust, don't call this function. */
- Assert(nappinfos >= 1 && appinfos != NULL);
+ /* If there's nothing to adjust, just return a duplication */
+ if (nappinfos == 0)
+ return copyObject(node);
/* Should never be translating a Query tree. */
Assert(node == NULL || !IsA(node, Query));
@@ -489,12 +491,10 @@ adjust_appendrel_attrs_multilevel(PlannerInfo *root, Node *node,
Relids child_relids,
Relids top_parent_relids)
{
- AppendRelInfo **appinfos;
- Bitmapset *parent_relids = NULL;
- int nappinfos;
- int cnt;
-
- Assert(bms_num_members(child_relids) == bms_num_members(top_parent_relids));
+ AppendRelInfo **appinfos;
+ Relids parent_relids = NULL;
+ int nappinfos;
+ int cnt;
appinfos = find_appinfos_by_relids(root, child_relids, &nappinfos);
@@ -506,8 +506,13 @@ adjust_appendrel_attrs_multilevel(PlannerInfo *root, Node *node,
parent_relids = bms_add_member(parent_relids, appinfo->parent_relid);
}
- /* Recurse if immediate parent is not the top parent. */
- if (!bms_equal(parent_relids, top_parent_relids))
+ /*
+ * Recurse if immediate parent is not the top parent. Keep in mind that in a
+ * case of asymmetric JOIN top_parent_relids can contain relids which aren't
+ * part of an append node.
+ */
+ if (!bms_equal(parent_relids, top_parent_relids) &&
+ !bms_is_subset(parent_relids, top_parent_relids))
node = adjust_appendrel_attrs_multilevel(root, node, parent_relids,
top_parent_relids);
@@ -515,6 +520,7 @@ adjust_appendrel_attrs_multilevel(PlannerInfo *root, Node *node,
node = adjust_appendrel_attrs(root, node, nappinfos, appinfos);
pfree(appinfos);
+ pfree(parent_relids);
return node;
}
@@ -565,6 +571,7 @@ adjust_child_relids_multilevel(PlannerInfo *root, Relids relids,
AppendRelInfo **appinfos;
int nappinfos;
Relids parent_relids = NULL;
+ Relids normal_relids = NULL;
Relids result;
Relids tmp_result = NULL;
int cnt;
@@ -579,12 +586,17 @@ adjust_child_relids_multilevel(PlannerInfo *root, Relids relids,
appinfos = find_appinfos_by_relids(root, child_relids, &nappinfos);
/* Construct relids set for the immediate parent of the given child. */
+ normal_relids = bms_copy(child_relids);
for (cnt = 0; cnt < nappinfos; cnt++)
{
AppendRelInfo *appinfo = appinfos[cnt];
parent_relids = bms_add_member(parent_relids, appinfo->parent_relid);
+ normal_relids = bms_del_member(normal_relids, appinfo->child_relid);
}
+ parent_relids = bms_union(parent_relids, normal_relids);
+ if (normal_relids)
+ bms_free(normal_relids);
/* Recurse if immediate parent is not the top parent. */
if (!bms_equal(parent_relids, top_parent_relids))
@@ -715,11 +727,11 @@ AppendRelInfo **
find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos)
{
AppendRelInfo **appinfos;
+ int nrooms = bms_num_members(relids);
int cnt = 0;
int i;
- *nappinfos = bms_num_members(relids);
- appinfos = (AppendRelInfo **) palloc(sizeof(AppendRelInfo *) * *nappinfos);
+ appinfos = (AppendRelInfo **) palloc(sizeof(AppendRelInfo *) * nrooms);
i = -1;
while ((i = bms_next_member(relids, i)) >= 0)
@@ -727,10 +739,11 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos)
AppendRelInfo *appinfo = root->append_rel_array[i];
if (!appinfo)
- elog(ERROR, "child rel %d not found in append_rel_array", i);
+ continue;
appinfos[cnt++] = appinfo;
}
+ *nappinfos = cnt;
return appinfos;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index 9ce5f95e3b..03c0640473 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -4226,7 +4226,14 @@ do { \
MemoryContextSwitchTo(oldcontext);
}
- bms_free(required_outer);
+
+ /*
+ * If adjust_child_relids_multilevel don't do replacements it returns
+ * the original set, not a copy. It is possible in the case of asymmetric
+ * JOIN and child_rel->relids contains relids only of plane relations.
+ */
+ if (required_outer != old_ppi->ppi_req_outer)
+ bms_free(required_outer);
new_path->param_info = new_ppi;
diff --git a/src/backend/optimizer/util/relnode.c b/src/backend/optimizer/util/relnode.c
index e105a4d5f1..bb15fb2716 100644
--- a/src/backend/optimizer/util/relnode.c
+++ b/src/backend/optimizer/util/relnode.c
@@ -790,11 +790,8 @@ build_child_join_rel(PlannerInfo *root, RelOptInfo *outer_rel,
AppendRelInfo **appinfos;
int nappinfos;
- /* Only joins between "other" relations land here. */
- Assert(IS_OTHER_REL(outer_rel) && IS_OTHER_REL(inner_rel));
-
- /* The parent joinrel should have consider_partitionwise_join set. */
- Assert(parent_joinrel->consider_partitionwise_join);
+ /* Either of relations must be "other" relation at least. */
+ Assert(IS_OTHER_REL(outer_rel) || IS_OTHER_REL(inner_rel));
joinrel->reloptkind = RELOPT_OTHER_JOINREL;
joinrel->relids = bms_union(outer_rel->relids, inner_rel->relids);
@@ -851,8 +848,11 @@ build_child_join_rel(PlannerInfo *root, RelOptInfo *outer_rel,
joinrel->partexprs = NULL;
joinrel->nullable_partexprs = NULL;
- joinrel->top_parent_relids = bms_union(outer_rel->top_parent_relids,
- inner_rel->top_parent_relids);
+ joinrel->top_parent_relids =
+ bms_union(IS_OTHER_REL(outer_rel) ?
+ outer_rel->top_parent_relids : outer_rel->relids,
+ IS_OTHER_REL(inner_rel) ?
+ inner_rel->top_parent_relids : inner_rel->relids);
/* Compute information relevant to foreign relations. */
set_foreign_rel_properties(joinrel, outer_rel, inner_rel);
@@ -2033,9 +2033,8 @@ build_child_join_reltarget(PlannerInfo *root,
{
/* Build the targetlist */
childrel->reltarget->exprs = (List *)
- adjust_appendrel_attrs(root,
- (Node *) parentrel->reltarget->exprs,
- nappinfos, appinfos);
+ adjust_appendrel_attrs_multilevel(root, (Node *)parentrel->reltarget->exprs,
+ childrel->relids, parentrel->relids);
/* Set the cost and width fields */
childrel->reltarget->cost.startup = parentrel->reltarget->cost.startup;
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index f1d111063c..a0106dc539 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -109,7 +109,12 @@ extern bool have_join_order_restriction(PlannerInfo *root,
extern bool have_dangerous_phv(PlannerInfo *root,
Relids outer_relids, Relids inner_params);
extern void mark_dummy_rel(RelOptInfo *rel);
-
+extern void try_asymmetric_partitionwise_join(PlannerInfo *root,
+ RelOptInfo *joinrel,
+ RelOptInfo *outer_rel,
+ RelOptInfo *inner_rel,
+ JoinType jointype,
+ JoinPathExtraData *extra);
/*
* equivclass.c
* routines for managing EquivalenceClasses
diff --git a/src/test/regress/expected/partition_join.out b/src/test/regress/expected/partition_join.out
index 27f7525b3e..601f08662e 100644
--- a/src/test/regress/expected/partition_join.out
+++ b/src/test/regress/expected/partition_join.out
@@ -2320,6 +2320,312 @@ SELECT t1.a, t1.c, t2.b, t2.c FROM prt1_adv t1 INNER JOIN prt2_adv t2 ON (t1.a =
375 | 0375 | 375 | 0375
(8 rows)
+--
+-- For asymmetric partition-wise join
+--
+CREATE TABLE prt5 (hkey int, a int, b int) PARTITION BY HASH(hkey);
+CREATE TABLE prt5_p0 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 0);
+CREATE TABLE prt5_p1 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 1);
+CREATE TABLE prt5_p2 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 2);
+CREATE TABLE prt6 (aid int, alabel text) PARTITION BY HASH(aid);
+CREATE TABLE prt6_p0 PARTITION OF prt6
+ FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE prt6_p1 PARTITION OF prt6
+ FOR VALUES WITH (modulus 2, remainder 1);
+CREATE TABLE t5_1 (aid int, alabel text);
+CREATE TABLE t5_2 (bid int, blabel text);
+INSERT INTO prt5 (SELECT x, (1000.0 * random())::int,
+ (1000.0 * random())::int
+ FROM generate_series(1,1000000) x);
+INSERT INTO t5_1 (SELECT x, md5(x::text) FROM generate_series(-200, 800) x);
+INSERT INTO t5_2 (SELECT x, md5(x::text) FROM generate_series(-200, 800) x);
+INSERT INTO prt6 (SELECT * FROM t5_1);
+VACUUM ANALYZE prt5,prt6,t5_1,t5_2;
+SET max_parallel_workers_per_gather = 0;
+-- Trivial asymmetric JOIN of partitioned table with a relation
+EXPLAIN (COSTS OFF)
+SELECT *
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+-------------------------------------------------------
+ Append
+ -> Hash Join
+ Hash Cond: (prt5_1.a = t5_1.aid)
+ -> Seq Scan on prt5_p0 prt5_1
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_2.a = t5_1.aid)
+ -> Seq Scan on prt5_p1 prt5_2
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_3.a = t5_1.aid)
+ -> Seq Scan on prt5_p2 prt5_3
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+(19 rows)
+
+-- The same, but appended with UNION ALL
+EXPLAIN (COSTS OFF)
+SELECT * FROM (
+ (SELECT * FROM prt5_p0)
+ UNION ALL
+ (SELECT * FROM prt5_p1)
+ UNION ALL
+ (SELECT * FROM prt5_p2)
+ ) AS sq1
+JOIN t5_1 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+-------------------------------------------------------
+ Append
+ -> Hash Join
+ Hash Cond: (prt5_p0.a = t5_1.aid)
+ -> Seq Scan on prt5_p0
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_p1.a = t5_1.aid)
+ -> Seq Scan on prt5_p1
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_p2.a = t5_1.aid)
+ -> Seq Scan on prt5_p2
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+(19 rows)
+
+-- Don't allow asymmetric JOIN of two partitioned tables.
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 JOIN prt6 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+-------------------------------------------------------
+ Hash Join
+ Hash Cond: (prt5.a = prt6.aid)
+ -> Append
+ -> Seq Scan on prt5_p0 prt5_1
+ -> Seq Scan on prt5_p1 prt5_2
+ -> Seq Scan on prt5_p2 prt5_3
+ -> Hash
+ -> Append
+ -> Seq Scan on prt6_p0 prt6_1
+ Filter: (alabel ~~ '%abc%'::text)
+ -> Seq Scan on prt6_p1 prt6_2
+ Filter: (alabel ~~ '%abc%'::text)
+(12 rows)
+
+-- Check asymmetric JOIN with Subquery
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 JOIN (
+ SELECT * FROM prt6 LIMIT 1000
+) AS sq1 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+----------------------------------------------------------------
+ Append
+ -> Hash Join
+ Hash Cond: (prt5_1.a = sq1.aid)
+ -> Seq Scan on prt5_p0 prt5_1
+ -> Hash
+ -> Subquery Scan on sq1
+ Filter: (sq1.alabel ~~ '%abc%'::text)
+ -> Limit
+ -> Append
+ -> Seq Scan on prt6_p0 prt6_1
+ -> Seq Scan on prt6_p1 prt6_2
+ -> Hash Join
+ Hash Cond: (prt5_2.a = sq1.aid)
+ -> Seq Scan on prt5_p1 prt5_2
+ -> Hash
+ -> Subquery Scan on sq1
+ Filter: (sq1.alabel ~~ '%abc%'::text)
+ -> Limit
+ -> Append
+ -> Seq Scan on prt6_p0 prt6_4
+ -> Seq Scan on prt6_p1 prt6_5
+ -> Hash Join
+ Hash Cond: (prt5_3.a = sq1.aid)
+ -> Seq Scan on prt5_p2 prt5_3
+ -> Hash
+ -> Subquery Scan on sq1
+ Filter: (sq1.alabel ~~ '%abc%'::text)
+ -> Limit
+ -> Append
+ -> Seq Scan on prt6_p0 prt6_7
+ -> Seq Scan on prt6_p1 prt6_8
+(31 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT *
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+ QUERY PLAN
+------------------------------------------------------------
+ Append
+ -> Hash Join
+ Hash Cond: (prt5_1.a = t5_1.aid)
+ -> Hash Join
+ Hash Cond: (prt5_1.b = t5_2.bid)
+ -> Seq Scan on prt5_p0 prt5_1
+ -> Hash
+ -> Seq Scan on t5_2
+ Filter: (blabel ~~ '%cd%'::text)
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%ab%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_2.a = t5_1.aid)
+ -> Hash Join
+ Hash Cond: (prt5_2.b = t5_2.bid)
+ -> Seq Scan on prt5_p1 prt5_2
+ -> Hash
+ -> Seq Scan on t5_2
+ Filter: (blabel ~~ '%cd%'::text)
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%ab%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_3.a = t5_1.aid)
+ -> Hash Join
+ Hash Cond: (prt5_3.b = t5_2.bid)
+ -> Seq Scan on prt5_p2 prt5_3
+ -> Hash
+ -> Seq Scan on t5_2
+ Filter: (blabel ~~ '%cd%'::text)
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%ab%'::text)
+(34 rows)
+
+-- unable to extract non-partitioned right relation
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 RIGHT JOIN t5_1 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+-----------------------------------------------
+ Hash Right Join
+ Hash Cond: (prt5.a = t5_1.aid)
+ Join Filter: (t5_1.alabel ~~ '%abc%'::text)
+ -> Append
+ -> Seq Scan on prt5_p0 prt5_1
+ -> Seq Scan on prt5_p1 prt5_2
+ -> Seq Scan on prt5_p2 prt5_3
+ -> Hash
+ -> Seq Scan on t5_1
+(9 rows)
+
+-- left side can be extracted, but no cost benefit
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 LEFT JOIN t5_1 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+-------------------------------------------------
+ Hash Left Join
+ Hash Cond: (prt5.a = t5_1.aid)
+ -> Append
+ -> Seq Scan on prt5_p0 prt5_1
+ -> Seq Scan on prt5_p1 prt5_2
+ -> Seq Scan on prt5_p2 prt5_3
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+(9 rows)
+
+-- validation of the results with/without asymmetric partition-wise join
+SELECT * INTO pg_temp.result01a
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+SELECT * INTO pg_temp.result02a
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+SET enable_partitionwise_join = off;
+SELECT * INTO pg_temp.result01b
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+SELECT * INTO pg_temp.result02b
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+SELECT * FROM pg_temp.result01a EXCEPT SELECT * FROM pg_temp.result01b;
+ hkey | a | b | aid | alabel
+------+---+---+-----+--------
+(0 rows)
+
+SELECT * FROM pg_temp.result02a EXCEPT SELECT * FROM pg_temp.result02b;
+ hkey | a | b | aid | alabel | bid | blabel
+------+---+---+-----+--------+-----+--------
+(0 rows)
+
+RESET max_parallel_workers_per_gather;
+SET enable_partitionwise_join = true;
+-- Parameterized path examples.
+CREATE TABLE prta (id integer, payload text) PARTITION BY HASH (id);
+CREATE TABLE prta1 PARTITION OF prta FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE prta2 PARTITION OF prta FOR VALUES WITH (modulus 2, remainder 1);
+CREATE UNIQUE INDEX ON prta1 (id);
+CREATE UNIQUE INDEX ON prta2 (id);
+INSERT INTO prta (id, payload)
+ (SELECT *, ('abc' || id)::text AS payload
+ FROM generate_series(1,1) AS id);
+CREATE TABLE prtb (id integer, payload text) PARTITION BY HASH (id);
+CREATE TABLE prtb1 PARTITION OF prtb FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE prtb2 PARTITION OF prtb FOR VALUES WITH (modulus 2, remainder 1);
+CREATE UNIQUE INDEX ON prtb1 (id);
+CREATE UNIQUE INDEX ON prtb2 (id);
+INSERT INTO prtb (id, payload)
+ (SELECT *, ('def' || id)::text AS payload
+ FROM generate_series(1,1000) AS id);
+CREATE TABLE e (id integer, payload text) PARTITION BY HASH (id);
+CREATE TABLE e1 PARTITION OF e FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE e2 PARTITION OF e FOR VALUES WITH (modulus 2, remainder 1);
+INSERT INTO e (id, payload)
+ (SELECT *, ('ghi' || id)::text AS payload
+ FROM generate_series(1,1000) AS id);
+CREATE UNIQUE INDEX ON e1 (id);
+CREATE UNIQUE INDEX ON e2 (id);
+ANALYZE prta,prtb,e;
+EXPLAIN (COSTS OFF)
+SELECT * FROM prta,prtb WHERE prta.id=prtb.id;
+ QUERY PLAN
+-----------------------------------------------------------
+ Append
+ -> Nested Loop
+ -> Seq Scan on prta1 prta_1
+ -> Index Scan using prtb1_id_idx on prtb1 prtb_1
+ Index Cond: (id = prta_1.id)
+ -> Nested Loop
+ -> Seq Scan on prta2 prta_2
+ -> Index Scan using prtb2_id_idx on prtb2 prtb_2
+ Index Cond: (id = prta_2.id)
+(9 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT * FROM prta,prtb,e WHERE prta.id=prtb.id AND prta.id=e.id;
+ QUERY PLAN
+-----------------------------------------------------------------
+ Append
+ -> Nested Loop
+ Join Filter: (prta_1.id = e_1.id)
+ -> Nested Loop
+ -> Seq Scan on prta1 prta_1
+ -> Index Scan using prtb1_id_idx on prtb1 prtb_1
+ Index Cond: (id = prta_1.id)
+ -> Index Scan using e1_id_idx on e1 e_1
+ Index Cond: (id = prtb_1.id)
+ -> Nested Loop
+ Join Filter: (prta_2.id = e_2.id)
+ -> Nested Loop
+ -> Seq Scan on prta2 prta_2
+ -> Index Scan using prtb2_id_idx on prtb2 prtb_2
+ Index Cond: (id = prta_2.id)
+ -> Index Scan using e2_id_idx on e2 e_2
+ Index Cond: (id = prtb_2.id)
+(17 rows)
+
-- semi join
EXPLAIN (COSTS OFF)
SELECT t1.* FROM prt1_adv t1 WHERE EXISTS (SELECT 1 FROM prt2_adv t2 WHERE t1.a = t2.b) AND t1.b = 0 ORDER BY t1.a;
diff --git a/src/test/regress/sql/partition_join.sql b/src/test/regress/sql/partition_join.sql
index d97b5b69ff..51b81ce3f8 100644
--- a/src/test/regress/sql/partition_join.sql
+++ b/src/test/regress/sql/partition_join.sql
@@ -536,6 +536,132 @@ EXPLAIN (COSTS OFF)
SELECT t1.a, t1.c, t2.b, t2.c FROM prt1_adv t1 INNER JOIN prt2_adv t2 ON (t1.a = t2.b) WHERE t1.b = 0 ORDER BY t1.a, t2.b;
SELECT t1.a, t1.c, t2.b, t2.c FROM prt1_adv t1 INNER JOIN prt2_adv t2 ON (t1.a = t2.b) WHERE t1.b = 0 ORDER BY t1.a, t2.b;
+--
+-- For asymmetric partition-wise join
+--
+CREATE TABLE prt5 (hkey int, a int, b int) PARTITION BY HASH(hkey);
+CREATE TABLE prt5_p0 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 0);
+CREATE TABLE prt5_p1 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 1);
+CREATE TABLE prt5_p2 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 2);
+CREATE TABLE prt6 (aid int, alabel text) PARTITION BY HASH(aid);
+CREATE TABLE prt6_p0 PARTITION OF prt6
+ FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE prt6_p1 PARTITION OF prt6
+ FOR VALUES WITH (modulus 2, remainder 1);
+CREATE TABLE t5_1 (aid int, alabel text);
+CREATE TABLE t5_2 (bid int, blabel text);
+
+INSERT INTO prt5 (SELECT x, (1000.0 * random())::int,
+ (1000.0 * random())::int
+ FROM generate_series(1,1000000) x);
+INSERT INTO t5_1 (SELECT x, md5(x::text) FROM generate_series(-200, 800) x);
+INSERT INTO t5_2 (SELECT x, md5(x::text) FROM generate_series(-200, 800) x);
+INSERT INTO prt6 (SELECT * FROM t5_1);
+
+VACUUM ANALYZE prt5,prt6,t5_1,t5_2;
+
+SET max_parallel_workers_per_gather = 0;
+
+-- Trivial asymmetric JOIN of partitioned table with a relation
+EXPLAIN (COSTS OFF)
+SELECT *
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+
+-- The same, but appended with UNION ALL
+EXPLAIN (COSTS OFF)
+SELECT * FROM (
+ (SELECT * FROM prt5_p0)
+ UNION ALL
+ (SELECT * FROM prt5_p1)
+ UNION ALL
+ (SELECT * FROM prt5_p2)
+ ) AS sq1
+JOIN t5_1 ON a = aid AND alabel like '%abc%';
+
+-- Don't allow asymmetric JOIN of two partitioned tables.
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 JOIN prt6 ON a = aid AND alabel like '%abc%';
+
+-- Check asymmetric JOIN with Subquery
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 JOIN (
+ SELECT * FROM prt6 LIMIT 1000
+) AS sq1 ON a = aid AND alabel like '%abc%';
+
+EXPLAIN (COSTS OFF)
+SELECT *
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+
+-- unable to extract non-partitioned right relation
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 RIGHT JOIN t5_1 ON a = aid AND alabel like '%abc%';
+-- left side can be extracted, but no cost benefit
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 LEFT JOIN t5_1 ON a = aid AND alabel like '%abc%';
+
+-- validation of the results with/without asymmetric partition-wise join
+SELECT * INTO pg_temp.result01a
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+
+SELECT * INTO pg_temp.result02a
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+
+SET enable_partitionwise_join = off;
+
+SELECT * INTO pg_temp.result01b
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+
+SELECT * INTO pg_temp.result02b
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+
+SELECT * FROM pg_temp.result01a EXCEPT SELECT * FROM pg_temp.result01b;
+SELECT * FROM pg_temp.result02a EXCEPT SELECT * FROM pg_temp.result02b;
+
+RESET max_parallel_workers_per_gather;
+SET enable_partitionwise_join = true;
+
+-- Parameterized path examples.
+CREATE TABLE prta (id integer, payload text) PARTITION BY HASH (id);
+CREATE TABLE prta1 PARTITION OF prta FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE prta2 PARTITION OF prta FOR VALUES WITH (modulus 2, remainder 1);
+CREATE UNIQUE INDEX ON prta1 (id);
+CREATE UNIQUE INDEX ON prta2 (id);
+INSERT INTO prta (id, payload)
+ (SELECT *, ('abc' || id)::text AS payload
+ FROM generate_series(1,1) AS id);
+
+CREATE TABLE prtb (id integer, payload text) PARTITION BY HASH (id);
+CREATE TABLE prtb1 PARTITION OF prtb FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE prtb2 PARTITION OF prtb FOR VALUES WITH (modulus 2, remainder 1);
+CREATE UNIQUE INDEX ON prtb1 (id);
+CREATE UNIQUE INDEX ON prtb2 (id);
+INSERT INTO prtb (id, payload)
+ (SELECT *, ('def' || id)::text AS payload
+ FROM generate_series(1,1000) AS id);
+
+CREATE TABLE e (id integer, payload text) PARTITION BY HASH (id);
+CREATE TABLE e1 PARTITION OF e FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE e2 PARTITION OF e FOR VALUES WITH (modulus 2, remainder 1);
+INSERT INTO e (id, payload)
+ (SELECT *, ('ghi' || id)::text AS payload
+ FROM generate_series(1,1000) AS id);
+CREATE UNIQUE INDEX ON e1 (id);
+CREATE UNIQUE INDEX ON e2 (id);
+
+ANALYZE prta,prtb,e;
+
+EXPLAIN (COSTS OFF)
+SELECT * FROM prta,prtb WHERE prta.id=prtb.id;
+
+EXPLAIN (COSTS OFF)
+SELECT * FROM prta,prtb,e WHERE prta.id=prtb.id AND prta.id=e.id;
+
-- semi join
EXPLAIN (COSTS OFF)
SELECT t1.* FROM prt1_adv t1 WHERE EXISTS (SELECT 1 FROM prt2_adv t2 WHERE t1.a = t2.b) AND t1.b = 0 ORDER BY t1.a;
--
2.31.1
Attachments:
[text/plain] v15-0001-Asymmetric-partitionwise-join.patch (32.9K, ../../[email protected]/2-v15-0001-Asymmetric-partitionwise-join.patch)
download | inline diff:
From d653b4f65486e2cefafda61811390c4095eae371 Mon Sep 17 00:00:00 2001
From: Andrey Lepikhov <[email protected]>
Date: Fri, 2 Apr 2021 11:02:20 +0500
Subject: [PATCH] Asymmetric partitionwise join.
Teach optimizer to consider partitionwise join of non-partitioned
table with each partition of partitioned table.
Disallow asymmetric machinery for joining of two partitioned (or appended)
relations because it could cause CPU and memory huge consumption
during reparameterization of NestLoop path.
---
src/backend/optimizer/path/joinpath.c | 9 +
src/backend/optimizer/path/joinrels.c | 172 +++++++++++
src/backend/optimizer/plan/setrefs.c | 17 +-
src/backend/optimizer/util/appendinfo.c | 37 ++-
src/backend/optimizer/util/pathnode.c | 9 +-
src/backend/optimizer/util/relnode.c | 19 +-
src/include/optimizer/paths.h | 7 +-
src/test/regress/expected/partition_join.out | 306 +++++++++++++++++++
src/test/regress/sql/partition_join.sql | 126 ++++++++
9 files changed, 672 insertions(+), 30 deletions(-)
diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c
index b67b517770..7a1a7b2e86 100644
--- a/src/backend/optimizer/path/joinpath.c
+++ b/src/backend/optimizer/path/joinpath.c
@@ -335,6 +335,15 @@ add_paths_to_joinrel(PlannerInfo *root,
if (set_join_pathlist_hook)
set_join_pathlist_hook(root, joinrel, outerrel, innerrel,
jointype, &extra);
+
+ /*
+ * 7. If outer relation is delivered from partition-tables, consider
+ * distributing inner relation to every partition-leaf prior to
+ * append these leafs.
+ */
+ try_asymmetric_partitionwise_join(root, joinrel,
+ outerrel, innerrel,
+ jointype, &extra);
}
/*
diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c
index 0dbe2ac726..2839b7acc3 100644
--- a/src/backend/optimizer/path/joinrels.c
+++ b/src/backend/optimizer/path/joinrels.c
@@ -16,6 +16,7 @@
#include "miscadmin.h"
#include "optimizer/appendinfo.h"
+#include "optimizer/cost.h"
#include "optimizer/joininfo.h"
#include "optimizer/pathnode.h"
#include "optimizer/paths.h"
@@ -1551,6 +1552,177 @@ try_partitionwise_join(PlannerInfo *root, RelOptInfo *rel1, RelOptInfo *rel2,
}
}
+/*
+ * Build RelOptInfo on JOIN of each partition of the outer relation and the inner
+ * relation. Return List of such RelOptInfo's. Return NIL, if at least one of
+ * these JOINs are impossible to build.
+ */
+static List *
+extract_asymmetric_partitionwise_subjoin(PlannerInfo *root,
+ RelOptInfo *joinrel,
+ AppendPath *append_path,
+ RelOptInfo *inner_rel,
+ JoinType jointype,
+ JoinPathExtraData *extra)
+{
+ List *result = NIL;
+ ListCell *lc;
+
+ foreach (lc, append_path->subpaths)
+ {
+ Path *child_path = lfirst(lc);
+ RelOptInfo *child_rel = child_path->parent;
+ Relids child_joinrelids;
+ Relids parent_relids;
+ RelOptInfo *child_joinrel;
+ SpecialJoinInfo *child_sjinfo;
+ List *child_restrictlist;
+
+ child_joinrelids = bms_union(child_rel->relids, inner_rel->relids);
+ parent_relids = bms_union(append_path->path.parent->relids,
+ inner_rel->relids);
+
+ child_sjinfo = build_child_join_sjinfo(root, extra->sjinfo,
+ child_rel->relids,
+ inner_rel->relids);
+ child_restrictlist = (List *)
+ adjust_appendrel_attrs_multilevel(root, (Node *)extra->restrictlist,
+ child_joinrelids, parent_relids);
+
+ child_joinrel = find_join_rel(root, child_joinrelids);
+ if (!child_joinrel)
+ child_joinrel = build_child_join_rel(root,
+ child_rel,
+ inner_rel,
+ joinrel,
+ child_restrictlist,
+ child_sjinfo,
+ jointype);
+ else
+ {
+ /*
+ * TODO:
+ * Can't imagine situation when join relation already exists. But in
+ * the 'partition_join' regression test it happens.
+ * It may be an indicator of possible problems.
+ */
+ }
+
+ populate_joinrel_with_paths(root,
+ child_rel,
+ inner_rel,
+ child_joinrel,
+ child_sjinfo,
+ child_restrictlist);
+
+ /* Give up if asymmetric partition-wise join is not available */
+ if (child_joinrel->pathlist == NIL)
+ return NIL;
+
+ set_cheapest(child_joinrel);
+ result = lappend(result, child_joinrel);
+ }
+ return result;
+}
+
+static bool
+is_asymmetric_join_capable(PlannerInfo *root,
+ RelOptInfo *outer_rel,
+ RelOptInfo *inner_rel,
+ JoinType jointype)
+{
+ ListCell *lc;
+
+ if (jointype != JOIN_INNER && jointype != JOIN_LEFT)
+ return false;
+
+ if (IS_OTHER_REL(outer_rel) || IS_OTHER_REL(inner_rel))
+ return false;
+
+ /* Disallow recursive usage of asymmertic join machinery */
+ if (root->join_rel_level == NULL)
+ return false;
+
+ /*
+ * Don't allow asymmetric JOIN of two append subplans.
+ * In the case of a parameterized NL join, a reparameterization procedure
+ * will lead to large memory allocations and a CPU consumption:
+ * each reparameterization will induce subpath duplication, creating new
+ * ParamPathInfo instance and increasing of ppilist up to number of
+ * partitions in the inner. Also, if we have many partitions, each bitmapset
+ * variable will be large and many leaks of such variable (caused by relid
+ * replacement) will highly increase memory consumption.
+ * So, we deny such paths for now.
+ */
+ foreach(lc, inner_rel->pathlist)
+ {
+ if (IsA(lfirst(lc), AppendPath))
+ return false;
+ }
+
+ return true;
+}
+
+void
+try_asymmetric_partitionwise_join(PlannerInfo *root,
+ RelOptInfo *joinrel,
+ RelOptInfo *outer_rel,
+ RelOptInfo *inner_rel,
+ JoinType jointype,
+ JoinPathExtraData *extra)
+{
+ ListCell *lc;
+
+ /*
+ * Try this kind of paths if we allow complex partitionwise joins and we know
+ * we can build this join safely.
+ */
+ if (!enable_partitionwise_join ||
+ !is_asymmetric_join_capable(root, outer_rel, inner_rel, jointype))
+ return;
+
+ foreach (lc, outer_rel->pathlist)
+ {
+ AppendPath *append_path = lfirst(lc);
+
+ /*
+ * We assume this pathlist keeps at least one AppendPath that
+ * represents partitioned table-scan, symmetric or asymmetric
+ * partition-wise join.
+ */
+ if (IsA(append_path, AppendPath))
+ {
+ List **join_rel_level_saved;
+ List *live_childrels = NIL;
+
+ join_rel_level_saved = root->join_rel_level;
+ PG_TRY();
+ {
+ /* temporary disables "dynamic programming" algorithm */
+ root->join_rel_level = NULL;
+
+ live_childrels =
+ extract_asymmetric_partitionwise_subjoin(root,
+ joinrel,
+ append_path,
+ inner_rel,
+ jointype,
+ extra);
+ }
+ PG_FINALLY();
+ {
+ root->join_rel_level = join_rel_level_saved;
+ }
+ PG_END_TRY();
+
+ if (live_childrels != NIL)
+ add_paths_to_append_rel(root, joinrel, live_childrels);
+
+ break;
+ }
+ }
+}
+
/*
* Construct the SpecialJoinInfo for a child-join by translating
* SpecialJoinInfo for the join between parents. left_relids and right_relids
diff --git a/src/backend/optimizer/plan/setrefs.c b/src/backend/optimizer/plan/setrefs.c
index 61ccfd300b..834843ba33 100644
--- a/src/backend/optimizer/plan/setrefs.c
+++ b/src/backend/optimizer/plan/setrefs.c
@@ -281,24 +281,29 @@ set_plan_references(PlannerInfo *root, Plan *plan)
/*
* Adjust RT indexes of AppendRelInfos and add to final appendrels list.
- * We assume the AppendRelInfos were built during planning and don't need
- * to be copied.
+ * The AppendRelInfos are copied, because as a part of a subplan its could
+ * be visited many times in the case of asymmetric join.
*/
foreach(lc, root->append_rel_list)
{
AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc);
+ AppendRelInfo *newappinfo;
+
+ /* flat copy is enough since all valuable fields are scalars */
+ newappinfo = (AppendRelInfo *) palloc(sizeof(AppendRelInfo));
+ memcpy(newappinfo, appinfo, sizeof(AppendRelInfo));
/* adjust RT indexes */
- appinfo->parent_relid += rtoffset;
- appinfo->child_relid += rtoffset;
+ newappinfo->parent_relid += rtoffset;
+ newappinfo->child_relid += rtoffset;
/*
* Rather than adjust the translated_vars entries, just drop 'em.
* Neither the executor nor EXPLAIN currently need that data.
*/
- appinfo->translated_vars = NIL;
+ newappinfo->translated_vars = NIL;
- glob->appendRelations = lappend(glob->appendRelations, appinfo);
+ glob->appendRelations = lappend(glob->appendRelations, newappinfo);
}
/* Now fix the Plan tree */
diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c
index af46f581ac..ec8dee99b0 100644
--- a/src/backend/optimizer/util/appendinfo.c
+++ b/src/backend/optimizer/util/appendinfo.c
@@ -202,7 +202,9 @@ adjust_appendrel_attrs(PlannerInfo *root, Node *node, int nappinfos,
context.appinfos = appinfos;
/* If there's nothing to adjust, don't call this function. */
- Assert(nappinfos >= 1 && appinfos != NULL);
+ /* If there's nothing to adjust, just return a duplication */
+ if (nappinfos == 0)
+ return copyObject(node);
/* Should never be translating a Query tree. */
Assert(node == NULL || !IsA(node, Query));
@@ -489,12 +491,10 @@ adjust_appendrel_attrs_multilevel(PlannerInfo *root, Node *node,
Relids child_relids,
Relids top_parent_relids)
{
- AppendRelInfo **appinfos;
- Bitmapset *parent_relids = NULL;
- int nappinfos;
- int cnt;
-
- Assert(bms_num_members(child_relids) == bms_num_members(top_parent_relids));
+ AppendRelInfo **appinfos;
+ Relids parent_relids = NULL;
+ int nappinfos;
+ int cnt;
appinfos = find_appinfos_by_relids(root, child_relids, &nappinfos);
@@ -506,8 +506,13 @@ adjust_appendrel_attrs_multilevel(PlannerInfo *root, Node *node,
parent_relids = bms_add_member(parent_relids, appinfo->parent_relid);
}
- /* Recurse if immediate parent is not the top parent. */
- if (!bms_equal(parent_relids, top_parent_relids))
+ /*
+ * Recurse if immediate parent is not the top parent. Keep in mind that in a
+ * case of asymmetric JOIN top_parent_relids can contain relids which aren't
+ * part of an append node.
+ */
+ if (!bms_equal(parent_relids, top_parent_relids) &&
+ !bms_is_subset(parent_relids, top_parent_relids))
node = adjust_appendrel_attrs_multilevel(root, node, parent_relids,
top_parent_relids);
@@ -515,6 +520,7 @@ adjust_appendrel_attrs_multilevel(PlannerInfo *root, Node *node,
node = adjust_appendrel_attrs(root, node, nappinfos, appinfos);
pfree(appinfos);
+ pfree(parent_relids);
return node;
}
@@ -565,6 +571,7 @@ adjust_child_relids_multilevel(PlannerInfo *root, Relids relids,
AppendRelInfo **appinfos;
int nappinfos;
Relids parent_relids = NULL;
+ Relids normal_relids = NULL;
Relids result;
Relids tmp_result = NULL;
int cnt;
@@ -579,12 +586,17 @@ adjust_child_relids_multilevel(PlannerInfo *root, Relids relids,
appinfos = find_appinfos_by_relids(root, child_relids, &nappinfos);
/* Construct relids set for the immediate parent of the given child. */
+ normal_relids = bms_copy(child_relids);
for (cnt = 0; cnt < nappinfos; cnt++)
{
AppendRelInfo *appinfo = appinfos[cnt];
parent_relids = bms_add_member(parent_relids, appinfo->parent_relid);
+ normal_relids = bms_del_member(normal_relids, appinfo->child_relid);
}
+ parent_relids = bms_union(parent_relids, normal_relids);
+ if (normal_relids)
+ bms_free(normal_relids);
/* Recurse if immediate parent is not the top parent. */
if (!bms_equal(parent_relids, top_parent_relids))
@@ -715,11 +727,11 @@ AppendRelInfo **
find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos)
{
AppendRelInfo **appinfos;
+ int nrooms = bms_num_members(relids);
int cnt = 0;
int i;
- *nappinfos = bms_num_members(relids);
- appinfos = (AppendRelInfo **) palloc(sizeof(AppendRelInfo *) * *nappinfos);
+ appinfos = (AppendRelInfo **) palloc(sizeof(AppendRelInfo *) * nrooms);
i = -1;
while ((i = bms_next_member(relids, i)) >= 0)
@@ -727,10 +739,11 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos)
AppendRelInfo *appinfo = root->append_rel_array[i];
if (!appinfo)
- elog(ERROR, "child rel %d not found in append_rel_array", i);
+ continue;
appinfos[cnt++] = appinfo;
}
+ *nappinfos = cnt;
return appinfos;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index 9ce5f95e3b..03c0640473 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -4226,7 +4226,14 @@ do { \
MemoryContextSwitchTo(oldcontext);
}
- bms_free(required_outer);
+
+ /*
+ * If adjust_child_relids_multilevel don't do replacements it returns
+ * the original set, not a copy. It is possible in the case of asymmetric
+ * JOIN and child_rel->relids contains relids only of plane relations.
+ */
+ if (required_outer != old_ppi->ppi_req_outer)
+ bms_free(required_outer);
new_path->param_info = new_ppi;
diff --git a/src/backend/optimizer/util/relnode.c b/src/backend/optimizer/util/relnode.c
index e105a4d5f1..bb15fb2716 100644
--- a/src/backend/optimizer/util/relnode.c
+++ b/src/backend/optimizer/util/relnode.c
@@ -790,11 +790,8 @@ build_child_join_rel(PlannerInfo *root, RelOptInfo *outer_rel,
AppendRelInfo **appinfos;
int nappinfos;
- /* Only joins between "other" relations land here. */
- Assert(IS_OTHER_REL(outer_rel) && IS_OTHER_REL(inner_rel));
-
- /* The parent joinrel should have consider_partitionwise_join set. */
- Assert(parent_joinrel->consider_partitionwise_join);
+ /* Either of relations must be "other" relation at least. */
+ Assert(IS_OTHER_REL(outer_rel) || IS_OTHER_REL(inner_rel));
joinrel->reloptkind = RELOPT_OTHER_JOINREL;
joinrel->relids = bms_union(outer_rel->relids, inner_rel->relids);
@@ -851,8 +848,11 @@ build_child_join_rel(PlannerInfo *root, RelOptInfo *outer_rel,
joinrel->partexprs = NULL;
joinrel->nullable_partexprs = NULL;
- joinrel->top_parent_relids = bms_union(outer_rel->top_parent_relids,
- inner_rel->top_parent_relids);
+ joinrel->top_parent_relids =
+ bms_union(IS_OTHER_REL(outer_rel) ?
+ outer_rel->top_parent_relids : outer_rel->relids,
+ IS_OTHER_REL(inner_rel) ?
+ inner_rel->top_parent_relids : inner_rel->relids);
/* Compute information relevant to foreign relations. */
set_foreign_rel_properties(joinrel, outer_rel, inner_rel);
@@ -2033,9 +2033,8 @@ build_child_join_reltarget(PlannerInfo *root,
{
/* Build the targetlist */
childrel->reltarget->exprs = (List *)
- adjust_appendrel_attrs(root,
- (Node *) parentrel->reltarget->exprs,
- nappinfos, appinfos);
+ adjust_appendrel_attrs_multilevel(root, (Node *)parentrel->reltarget->exprs,
+ childrel->relids, parentrel->relids);
/* Set the cost and width fields */
childrel->reltarget->cost.startup = parentrel->reltarget->cost.startup;
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index f1d111063c..a0106dc539 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -109,7 +109,12 @@ extern bool have_join_order_restriction(PlannerInfo *root,
extern bool have_dangerous_phv(PlannerInfo *root,
Relids outer_relids, Relids inner_params);
extern void mark_dummy_rel(RelOptInfo *rel);
-
+extern void try_asymmetric_partitionwise_join(PlannerInfo *root,
+ RelOptInfo *joinrel,
+ RelOptInfo *outer_rel,
+ RelOptInfo *inner_rel,
+ JoinType jointype,
+ JoinPathExtraData *extra);
/*
* equivclass.c
* routines for managing EquivalenceClasses
diff --git a/src/test/regress/expected/partition_join.out b/src/test/regress/expected/partition_join.out
index 27f7525b3e..601f08662e 100644
--- a/src/test/regress/expected/partition_join.out
+++ b/src/test/regress/expected/partition_join.out
@@ -2320,6 +2320,312 @@ SELECT t1.a, t1.c, t2.b, t2.c FROM prt1_adv t1 INNER JOIN prt2_adv t2 ON (t1.a =
375 | 0375 | 375 | 0375
(8 rows)
+--
+-- For asymmetric partition-wise join
+--
+CREATE TABLE prt5 (hkey int, a int, b int) PARTITION BY HASH(hkey);
+CREATE TABLE prt5_p0 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 0);
+CREATE TABLE prt5_p1 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 1);
+CREATE TABLE prt5_p2 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 2);
+CREATE TABLE prt6 (aid int, alabel text) PARTITION BY HASH(aid);
+CREATE TABLE prt6_p0 PARTITION OF prt6
+ FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE prt6_p1 PARTITION OF prt6
+ FOR VALUES WITH (modulus 2, remainder 1);
+CREATE TABLE t5_1 (aid int, alabel text);
+CREATE TABLE t5_2 (bid int, blabel text);
+INSERT INTO prt5 (SELECT x, (1000.0 * random())::int,
+ (1000.0 * random())::int
+ FROM generate_series(1,1000000) x);
+INSERT INTO t5_1 (SELECT x, md5(x::text) FROM generate_series(-200, 800) x);
+INSERT INTO t5_2 (SELECT x, md5(x::text) FROM generate_series(-200, 800) x);
+INSERT INTO prt6 (SELECT * FROM t5_1);
+VACUUM ANALYZE prt5,prt6,t5_1,t5_2;
+SET max_parallel_workers_per_gather = 0;
+-- Trivial asymmetric JOIN of partitioned table with a relation
+EXPLAIN (COSTS OFF)
+SELECT *
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+-------------------------------------------------------
+ Append
+ -> Hash Join
+ Hash Cond: (prt5_1.a = t5_1.aid)
+ -> Seq Scan on prt5_p0 prt5_1
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_2.a = t5_1.aid)
+ -> Seq Scan on prt5_p1 prt5_2
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_3.a = t5_1.aid)
+ -> Seq Scan on prt5_p2 prt5_3
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+(19 rows)
+
+-- The same, but appended with UNION ALL
+EXPLAIN (COSTS OFF)
+SELECT * FROM (
+ (SELECT * FROM prt5_p0)
+ UNION ALL
+ (SELECT * FROM prt5_p1)
+ UNION ALL
+ (SELECT * FROM prt5_p2)
+ ) AS sq1
+JOIN t5_1 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+-------------------------------------------------------
+ Append
+ -> Hash Join
+ Hash Cond: (prt5_p0.a = t5_1.aid)
+ -> Seq Scan on prt5_p0
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_p1.a = t5_1.aid)
+ -> Seq Scan on prt5_p1
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_p2.a = t5_1.aid)
+ -> Seq Scan on prt5_p2
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+(19 rows)
+
+-- Don't allow asymmetric JOIN of two partitioned tables.
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 JOIN prt6 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+-------------------------------------------------------
+ Hash Join
+ Hash Cond: (prt5.a = prt6.aid)
+ -> Append
+ -> Seq Scan on prt5_p0 prt5_1
+ -> Seq Scan on prt5_p1 prt5_2
+ -> Seq Scan on prt5_p2 prt5_3
+ -> Hash
+ -> Append
+ -> Seq Scan on prt6_p0 prt6_1
+ Filter: (alabel ~~ '%abc%'::text)
+ -> Seq Scan on prt6_p1 prt6_2
+ Filter: (alabel ~~ '%abc%'::text)
+(12 rows)
+
+-- Check asymmetric JOIN with Subquery
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 JOIN (
+ SELECT * FROM prt6 LIMIT 1000
+) AS sq1 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+----------------------------------------------------------------
+ Append
+ -> Hash Join
+ Hash Cond: (prt5_1.a = sq1.aid)
+ -> Seq Scan on prt5_p0 prt5_1
+ -> Hash
+ -> Subquery Scan on sq1
+ Filter: (sq1.alabel ~~ '%abc%'::text)
+ -> Limit
+ -> Append
+ -> Seq Scan on prt6_p0 prt6_1
+ -> Seq Scan on prt6_p1 prt6_2
+ -> Hash Join
+ Hash Cond: (prt5_2.a = sq1.aid)
+ -> Seq Scan on prt5_p1 prt5_2
+ -> Hash
+ -> Subquery Scan on sq1
+ Filter: (sq1.alabel ~~ '%abc%'::text)
+ -> Limit
+ -> Append
+ -> Seq Scan on prt6_p0 prt6_4
+ -> Seq Scan on prt6_p1 prt6_5
+ -> Hash Join
+ Hash Cond: (prt5_3.a = sq1.aid)
+ -> Seq Scan on prt5_p2 prt5_3
+ -> Hash
+ -> Subquery Scan on sq1
+ Filter: (sq1.alabel ~~ '%abc%'::text)
+ -> Limit
+ -> Append
+ -> Seq Scan on prt6_p0 prt6_7
+ -> Seq Scan on prt6_p1 prt6_8
+(31 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT *
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+ QUERY PLAN
+------------------------------------------------------------
+ Append
+ -> Hash Join
+ Hash Cond: (prt5_1.a = t5_1.aid)
+ -> Hash Join
+ Hash Cond: (prt5_1.b = t5_2.bid)
+ -> Seq Scan on prt5_p0 prt5_1
+ -> Hash
+ -> Seq Scan on t5_2
+ Filter: (blabel ~~ '%cd%'::text)
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%ab%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_2.a = t5_1.aid)
+ -> Hash Join
+ Hash Cond: (prt5_2.b = t5_2.bid)
+ -> Seq Scan on prt5_p1 prt5_2
+ -> Hash
+ -> Seq Scan on t5_2
+ Filter: (blabel ~~ '%cd%'::text)
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%ab%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_3.a = t5_1.aid)
+ -> Hash Join
+ Hash Cond: (prt5_3.b = t5_2.bid)
+ -> Seq Scan on prt5_p2 prt5_3
+ -> Hash
+ -> Seq Scan on t5_2
+ Filter: (blabel ~~ '%cd%'::text)
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%ab%'::text)
+(34 rows)
+
+-- unable to extract non-partitioned right relation
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 RIGHT JOIN t5_1 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+-----------------------------------------------
+ Hash Right Join
+ Hash Cond: (prt5.a = t5_1.aid)
+ Join Filter: (t5_1.alabel ~~ '%abc%'::text)
+ -> Append
+ -> Seq Scan on prt5_p0 prt5_1
+ -> Seq Scan on prt5_p1 prt5_2
+ -> Seq Scan on prt5_p2 prt5_3
+ -> Hash
+ -> Seq Scan on t5_1
+(9 rows)
+
+-- left side can be extracted, but no cost benefit
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 LEFT JOIN t5_1 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+-------------------------------------------------
+ Hash Left Join
+ Hash Cond: (prt5.a = t5_1.aid)
+ -> Append
+ -> Seq Scan on prt5_p0 prt5_1
+ -> Seq Scan on prt5_p1 prt5_2
+ -> Seq Scan on prt5_p2 prt5_3
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+(9 rows)
+
+-- validation of the results with/without asymmetric partition-wise join
+SELECT * INTO pg_temp.result01a
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+SELECT * INTO pg_temp.result02a
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+SET enable_partitionwise_join = off;
+SELECT * INTO pg_temp.result01b
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+SELECT * INTO pg_temp.result02b
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+SELECT * FROM pg_temp.result01a EXCEPT SELECT * FROM pg_temp.result01b;
+ hkey | a | b | aid | alabel
+------+---+---+-----+--------
+(0 rows)
+
+SELECT * FROM pg_temp.result02a EXCEPT SELECT * FROM pg_temp.result02b;
+ hkey | a | b | aid | alabel | bid | blabel
+------+---+---+-----+--------+-----+--------
+(0 rows)
+
+RESET max_parallel_workers_per_gather;
+SET enable_partitionwise_join = true;
+-- Parameterized path examples.
+CREATE TABLE prta (id integer, payload text) PARTITION BY HASH (id);
+CREATE TABLE prta1 PARTITION OF prta FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE prta2 PARTITION OF prta FOR VALUES WITH (modulus 2, remainder 1);
+CREATE UNIQUE INDEX ON prta1 (id);
+CREATE UNIQUE INDEX ON prta2 (id);
+INSERT INTO prta (id, payload)
+ (SELECT *, ('abc' || id)::text AS payload
+ FROM generate_series(1,1) AS id);
+CREATE TABLE prtb (id integer, payload text) PARTITION BY HASH (id);
+CREATE TABLE prtb1 PARTITION OF prtb FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE prtb2 PARTITION OF prtb FOR VALUES WITH (modulus 2, remainder 1);
+CREATE UNIQUE INDEX ON prtb1 (id);
+CREATE UNIQUE INDEX ON prtb2 (id);
+INSERT INTO prtb (id, payload)
+ (SELECT *, ('def' || id)::text AS payload
+ FROM generate_series(1,1000) AS id);
+CREATE TABLE e (id integer, payload text) PARTITION BY HASH (id);
+CREATE TABLE e1 PARTITION OF e FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE e2 PARTITION OF e FOR VALUES WITH (modulus 2, remainder 1);
+INSERT INTO e (id, payload)
+ (SELECT *, ('ghi' || id)::text AS payload
+ FROM generate_series(1,1000) AS id);
+CREATE UNIQUE INDEX ON e1 (id);
+CREATE UNIQUE INDEX ON e2 (id);
+ANALYZE prta,prtb,e;
+EXPLAIN (COSTS OFF)
+SELECT * FROM prta,prtb WHERE prta.id=prtb.id;
+ QUERY PLAN
+-----------------------------------------------------------
+ Append
+ -> Nested Loop
+ -> Seq Scan on prta1 prta_1
+ -> Index Scan using prtb1_id_idx on prtb1 prtb_1
+ Index Cond: (id = prta_1.id)
+ -> Nested Loop
+ -> Seq Scan on prta2 prta_2
+ -> Index Scan using prtb2_id_idx on prtb2 prtb_2
+ Index Cond: (id = prta_2.id)
+(9 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT * FROM prta,prtb,e WHERE prta.id=prtb.id AND prta.id=e.id;
+ QUERY PLAN
+-----------------------------------------------------------------
+ Append
+ -> Nested Loop
+ Join Filter: (prta_1.id = e_1.id)
+ -> Nested Loop
+ -> Seq Scan on prta1 prta_1
+ -> Index Scan using prtb1_id_idx on prtb1 prtb_1
+ Index Cond: (id = prta_1.id)
+ -> Index Scan using e1_id_idx on e1 e_1
+ Index Cond: (id = prtb_1.id)
+ -> Nested Loop
+ Join Filter: (prta_2.id = e_2.id)
+ -> Nested Loop
+ -> Seq Scan on prta2 prta_2
+ -> Index Scan using prtb2_id_idx on prtb2 prtb_2
+ Index Cond: (id = prta_2.id)
+ -> Index Scan using e2_id_idx on e2 e_2
+ Index Cond: (id = prtb_2.id)
+(17 rows)
+
-- semi join
EXPLAIN (COSTS OFF)
SELECT t1.* FROM prt1_adv t1 WHERE EXISTS (SELECT 1 FROM prt2_adv t2 WHERE t1.a = t2.b) AND t1.b = 0 ORDER BY t1.a;
diff --git a/src/test/regress/sql/partition_join.sql b/src/test/regress/sql/partition_join.sql
index d97b5b69ff..51b81ce3f8 100644
--- a/src/test/regress/sql/partition_join.sql
+++ b/src/test/regress/sql/partition_join.sql
@@ -536,6 +536,132 @@ EXPLAIN (COSTS OFF)
SELECT t1.a, t1.c, t2.b, t2.c FROM prt1_adv t1 INNER JOIN prt2_adv t2 ON (t1.a = t2.b) WHERE t1.b = 0 ORDER BY t1.a, t2.b;
SELECT t1.a, t1.c, t2.b, t2.c FROM prt1_adv t1 INNER JOIN prt2_adv t2 ON (t1.a = t2.b) WHERE t1.b = 0 ORDER BY t1.a, t2.b;
+--
+-- For asymmetric partition-wise join
+--
+CREATE TABLE prt5 (hkey int, a int, b int) PARTITION BY HASH(hkey);
+CREATE TABLE prt5_p0 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 0);
+CREATE TABLE prt5_p1 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 1);
+CREATE TABLE prt5_p2 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 2);
+CREATE TABLE prt6 (aid int, alabel text) PARTITION BY HASH(aid);
+CREATE TABLE prt6_p0 PARTITION OF prt6
+ FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE prt6_p1 PARTITION OF prt6
+ FOR VALUES WITH (modulus 2, remainder 1);
+CREATE TABLE t5_1 (aid int, alabel text);
+CREATE TABLE t5_2 (bid int, blabel text);
+
+INSERT INTO prt5 (SELECT x, (1000.0 * random())::int,
+ (1000.0 * random())::int
+ FROM generate_series(1,1000000) x);
+INSERT INTO t5_1 (SELECT x, md5(x::text) FROM generate_series(-200, 800) x);
+INSERT INTO t5_2 (SELECT x, md5(x::text) FROM generate_series(-200, 800) x);
+INSERT INTO prt6 (SELECT * FROM t5_1);
+
+VACUUM ANALYZE prt5,prt6,t5_1,t5_2;
+
+SET max_parallel_workers_per_gather = 0;
+
+-- Trivial asymmetric JOIN of partitioned table with a relation
+EXPLAIN (COSTS OFF)
+SELECT *
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+
+-- The same, but appended with UNION ALL
+EXPLAIN (COSTS OFF)
+SELECT * FROM (
+ (SELECT * FROM prt5_p0)
+ UNION ALL
+ (SELECT * FROM prt5_p1)
+ UNION ALL
+ (SELECT * FROM prt5_p2)
+ ) AS sq1
+JOIN t5_1 ON a = aid AND alabel like '%abc%';
+
+-- Don't allow asymmetric JOIN of two partitioned tables.
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 JOIN prt6 ON a = aid AND alabel like '%abc%';
+
+-- Check asymmetric JOIN with Subquery
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 JOIN (
+ SELECT * FROM prt6 LIMIT 1000
+) AS sq1 ON a = aid AND alabel like '%abc%';
+
+EXPLAIN (COSTS OFF)
+SELECT *
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+
+-- unable to extract non-partitioned right relation
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 RIGHT JOIN t5_1 ON a = aid AND alabel like '%abc%';
+-- left side can be extracted, but no cost benefit
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 LEFT JOIN t5_1 ON a = aid AND alabel like '%abc%';
+
+-- validation of the results with/without asymmetric partition-wise join
+SELECT * INTO pg_temp.result01a
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+
+SELECT * INTO pg_temp.result02a
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+
+SET enable_partitionwise_join = off;
+
+SELECT * INTO pg_temp.result01b
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+
+SELECT * INTO pg_temp.result02b
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+
+SELECT * FROM pg_temp.result01a EXCEPT SELECT * FROM pg_temp.result01b;
+SELECT * FROM pg_temp.result02a EXCEPT SELECT * FROM pg_temp.result02b;
+
+RESET max_parallel_workers_per_gather;
+SET enable_partitionwise_join = true;
+
+-- Parameterized path examples.
+CREATE TABLE prta (id integer, payload text) PARTITION BY HASH (id);
+CREATE TABLE prta1 PARTITION OF prta FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE prta2 PARTITION OF prta FOR VALUES WITH (modulus 2, remainder 1);
+CREATE UNIQUE INDEX ON prta1 (id);
+CREATE UNIQUE INDEX ON prta2 (id);
+INSERT INTO prta (id, payload)
+ (SELECT *, ('abc' || id)::text AS payload
+ FROM generate_series(1,1) AS id);
+
+CREATE TABLE prtb (id integer, payload text) PARTITION BY HASH (id);
+CREATE TABLE prtb1 PARTITION OF prtb FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE prtb2 PARTITION OF prtb FOR VALUES WITH (modulus 2, remainder 1);
+CREATE UNIQUE INDEX ON prtb1 (id);
+CREATE UNIQUE INDEX ON prtb2 (id);
+INSERT INTO prtb (id, payload)
+ (SELECT *, ('def' || id)::text AS payload
+ FROM generate_series(1,1000) AS id);
+
+CREATE TABLE e (id integer, payload text) PARTITION BY HASH (id);
+CREATE TABLE e1 PARTITION OF e FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE e2 PARTITION OF e FOR VALUES WITH (modulus 2, remainder 1);
+INSERT INTO e (id, payload)
+ (SELECT *, ('ghi' || id)::text AS payload
+ FROM generate_series(1,1000) AS id);
+CREATE UNIQUE INDEX ON e1 (id);
+CREATE UNIQUE INDEX ON e2 (id);
+
+ANALYZE prta,prtb,e;
+
+EXPLAIN (COSTS OFF)
+SELECT * FROM prta,prtb WHERE prta.id=prtb.id;
+
+EXPLAIN (COSTS OFF)
+SELECT * FROM prta,prtb,e WHERE prta.id=prtb.id AND prta.id=e.id;
+
-- semi join
EXPLAIN (COSTS OFF)
SELECT t1.* FROM prt1_adv t1 WHERE EXISTS (SELECT 1 FROM prt2_adv t2 WHERE t1.a = t2.b) AND t1.b = 0 ORDER BY t1.a;
--
2.31.1
^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: Asymmetric partition-wise JOIN
2019-08-12 06:03 Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-08-22 16:05 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-08-23 22:01 ` Re: Asymmetric partition-wise JOIN Thomas Munro <[email protected]>
2019-08-24 08:33 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-12-01 03:24 ` Re: Asymmetric partition-wise JOIN Michael Paquier <[email protected]>
2019-12-27 07:34 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2020-07-01 09:10 ` Re: Asymmetric partition-wise JOIN Daniel Gustafsson <[email protected]>
2020-08-21 06:02 ` Re: Asymmetric partition-wise JOIN Andrey V. Lepikhov <[email protected]>
2020-11-09 10:53 ` Re: Asymmetric partition-wise JOIN Anastasia Lubennikova <[email protected]>
2020-11-30 14:43 ` Re: Asymmetric partition-wise JOIN Anastasia Lubennikova <[email protected]>
2021-04-30 03:10 ` Re: Asymmetric partition-wise JOIN Andrey V. Lepikhov <[email protected]>
2021-05-27 04:27 ` Re: Asymmetric partition-wise JOIN Andrey Lepikhov <[email protected]>
2021-06-18 12:02 ` Re: Asymmetric partition-wise JOIN Alexander Pyhalov <[email protected]>
2021-07-05 09:57 ` Re: Asymmetric partition-wise JOIN Andrey Lepikhov <[email protected]>
@ 2021-07-05 20:15 ` Zhihong Yu <[email protected]>
2021-07-06 09:28 ` Re: Asymmetric partition-wise JOIN Andrey Lepikhov <[email protected]>
2021-07-15 06:32 ` Re: Asymmetric partition-wise JOIN Andrey Lepikhov <[email protected]>
0 siblings, 2 replies; 28+ messages in thread
From: Zhihong Yu @ 2021-07-05 20:15 UTC (permalink / raw)
To: Andrey Lepikhov <[email protected]>; +Cc: Alexander Pyhalov <[email protected]>; Anastasia Lubennikova <[email protected]>; Daniel Gustafsson <[email protected]>; Kohei KaiGai <[email protected]>; Michael Paquier <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers
On Mon, Jul 5, 2021 at 2:57 AM Andrey Lepikhov <[email protected]>
wrote:
> On 18/6/21 15:02, Alexander Pyhalov wrote:
> > Andrey Lepikhov писал 2021-05-27 07:27:
> >> Next version of the patch.
> >> For searching any problems I forced this patch during 'make check'
> >> tests. Some bugs were found and fixed.
> >
> > Hi.
> > I've tested this patch and haven't found issues, but I have some
> comments.
> Thank you for review!
> > Variable names - child_join_rel and child_join_relids seem to be
> > inconsistent with rest of the file
> fixed
>
> > When build_child_join_rel() can return NULL?
> Fixed
> > Missing word:
> > each bitmapset variable will large => each bitmapset variable will be
> large
> Fixed
> > What hook do you refer to?
> Removed> You've changed function to copy appinfo, so now comment is
> incorrect.
> Thanks, fixed> Do I understand correctly that now parent_relids also
> contains relids of
> > relations from 'global' inner relation, which we join to childs?
> Yes
>
> --
> regards,
> Andrey Lepikhov
> Postgres Professional
>
Hi,
relations because it could cause CPU and memory huge consumption
during reparameterization of NestLoop path.
CPU and memory huge consumption -> huge consumption of CPU and memory
+ * relation. Return List of such RelOptInfo's. Return NIL, if at least one
of
+ * these JOINs are impossible to build.
at least one of these JOINs are impossible to build. -> at least one
of these JOINs is impossible to build.
+ * Can't imagine situation when join relation already exists.
But in
+ * the 'partition_join' regression test it happens.
+ * It may be an indicator of possible problems.
Should a log be added in the above case ?
+is_asymmetric_join_capable(PlannerInfo *root,
is_asymmetric_join_capable -> is_asymmetric_join_feasible
Cheers
^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: Asymmetric partition-wise JOIN
2019-08-12 06:03 Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-08-22 16:05 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-08-23 22:01 ` Re: Asymmetric partition-wise JOIN Thomas Munro <[email protected]>
2019-08-24 08:33 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-12-01 03:24 ` Re: Asymmetric partition-wise JOIN Michael Paquier <[email protected]>
2019-12-27 07:34 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2020-07-01 09:10 ` Re: Asymmetric partition-wise JOIN Daniel Gustafsson <[email protected]>
2020-08-21 06:02 ` Re: Asymmetric partition-wise JOIN Andrey V. Lepikhov <[email protected]>
2020-11-09 10:53 ` Re: Asymmetric partition-wise JOIN Anastasia Lubennikova <[email protected]>
2020-11-30 14:43 ` Re: Asymmetric partition-wise JOIN Anastasia Lubennikova <[email protected]>
2021-04-30 03:10 ` Re: Asymmetric partition-wise JOIN Andrey V. Lepikhov <[email protected]>
2021-05-27 04:27 ` Re: Asymmetric partition-wise JOIN Andrey Lepikhov <[email protected]>
2021-06-18 12:02 ` Re: Asymmetric partition-wise JOIN Alexander Pyhalov <[email protected]>
2021-07-05 09:57 ` Re: Asymmetric partition-wise JOIN Andrey Lepikhov <[email protected]>
2021-07-05 20:15 ` Re: Asymmetric partition-wise JOIN Zhihong Yu <[email protected]>
@ 2021-07-06 09:28 ` Andrey Lepikhov <[email protected]>
2021-07-06 13:09 ` Re: Asymmetric partition-wise JOIN Alexander Pyhalov <[email protected]>
1 sibling, 1 reply; 28+ messages in thread
From: Andrey Lepikhov @ 2021-07-06 09:28 UTC (permalink / raw)
To: Zhihong Yu <[email protected]>; +Cc: Alexander Pyhalov <[email protected]>; Anastasia Lubennikova <[email protected]>; Daniel Gustafsson <[email protected]>; Kohei KaiGai <[email protected]>; Michael Paquier <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers
On 5/7/21 23:15, Zhihong Yu wrote:
> On Mon, Jul 5, 2021 at 2:57 AM Andrey Lepikhov
> <[email protected] <mailto:[email protected]>> wrote:
> + * Can't imagine situation when join relation already
> exists. But in
> + * the 'partition_join' regression test it happens.
> + * It may be an indicator of possible problems.
> Should a log be added in the above case ?
I made additional analysis of this branch of code. This situation can
happen in the case of one child or if we join two plane tables with
partitioned. Both situations are legal and I think we don't needed to
add any log message here.
Other mistakes were fixed.
--
regards,
Andrey Lepikhov
Postgres Professional
From b9d7a148ab51f297c98345127e975f864fe6ef92 Mon Sep 17 00:00:00 2001
From: Andrey Lepikhov <[email protected]>
Date: Fri, 2 Apr 2021 11:02:20 +0500
Subject: [PATCH] Asymmetric partitionwise join.
Teach optimizer to consider partitionwise join of non-partitioned
table with each partition of partitioned table.
Disallow asymmetric machinery for joining of two partitioned (or appended)
relations because it could cause huge consumption of CPU and memory
during reparameterization of NestLoop path.
---
src/backend/optimizer/path/joinpath.c | 9 +
src/backend/optimizer/path/joinrels.c | 172 +++++++++++
src/backend/optimizer/plan/setrefs.c | 17 +-
src/backend/optimizer/util/appendinfo.c | 37 ++-
src/backend/optimizer/util/pathnode.c | 9 +-
src/backend/optimizer/util/relnode.c | 19 +-
src/include/optimizer/paths.h | 7 +-
src/test/regress/expected/partition_join.out | 306 +++++++++++++++++++
src/test/regress/sql/partition_join.sql | 126 ++++++++
9 files changed, 672 insertions(+), 30 deletions(-)
diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c
index b67b517770..7a1a7b2e86 100644
--- a/src/backend/optimizer/path/joinpath.c
+++ b/src/backend/optimizer/path/joinpath.c
@@ -335,6 +335,15 @@ add_paths_to_joinrel(PlannerInfo *root,
if (set_join_pathlist_hook)
set_join_pathlist_hook(root, joinrel, outerrel, innerrel,
jointype, &extra);
+
+ /*
+ * 7. If outer relation is delivered from partition-tables, consider
+ * distributing inner relation to every partition-leaf prior to
+ * append these leafs.
+ */
+ try_asymmetric_partitionwise_join(root, joinrel,
+ outerrel, innerrel,
+ jointype, &extra);
}
/*
diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c
index 0dbe2ac726..e9cbff9709 100644
--- a/src/backend/optimizer/path/joinrels.c
+++ b/src/backend/optimizer/path/joinrels.c
@@ -16,6 +16,7 @@
#include "miscadmin.h"
#include "optimizer/appendinfo.h"
+#include "optimizer/cost.h"
#include "optimizer/joininfo.h"
#include "optimizer/pathnode.h"
#include "optimizer/paths.h"
@@ -1551,6 +1552,177 @@ try_partitionwise_join(PlannerInfo *root, RelOptInfo *rel1, RelOptInfo *rel2,
}
}
+/*
+ * Build RelOptInfo on JOIN of each partition of the outer relation and the inner
+ * relation. Return List of such RelOptInfo's. Return NIL, if at least one of
+ * these JOINs is impossible to build.
+ */
+static List *
+extract_asymmetric_partitionwise_subjoin(PlannerInfo *root,
+ RelOptInfo *joinrel,
+ AppendPath *append_path,
+ RelOptInfo *inner_rel,
+ JoinType jointype,
+ JoinPathExtraData *extra)
+{
+ List *result = NIL;
+ ListCell *lc;
+
+ foreach (lc, append_path->subpaths)
+ {
+ Path *child_path = lfirst(lc);
+ RelOptInfo *child_rel = child_path->parent;
+ Relids child_joinrelids;
+ Relids parent_relids;
+ RelOptInfo *child_joinrel;
+ SpecialJoinInfo *child_sjinfo;
+ List *child_restrictlist;
+
+ child_joinrelids = bms_union(child_rel->relids, inner_rel->relids);
+ parent_relids = bms_union(append_path->path.parent->relids,
+ inner_rel->relids);
+
+ child_sjinfo = build_child_join_sjinfo(root, extra->sjinfo,
+ child_rel->relids,
+ inner_rel->relids);
+ child_restrictlist = (List *)
+ adjust_appendrel_attrs_multilevel(root, (Node *)extra->restrictlist,
+ child_joinrelids, parent_relids);
+
+ child_joinrel = find_join_rel(root, child_joinrelids);
+ if (!child_joinrel)
+ child_joinrel = build_child_join_rel(root,
+ child_rel,
+ inner_rel,
+ joinrel,
+ child_restrictlist,
+ child_sjinfo,
+ jointype);
+ else
+ {
+ /*
+ * The join relation already exists. For example, it could happen if
+ * we join two plane tables with partitioned table(s). Do nothing.
+ */
+ }
+
+ populate_joinrel_with_paths(root,
+ child_rel,
+ inner_rel,
+ child_joinrel,
+ child_sjinfo,
+ child_restrictlist);
+
+ /* Give up if asymmetric partition-wise join is not available */
+ if (child_joinrel->pathlist == NIL)
+ return NIL;
+
+ set_cheapest(child_joinrel);
+ result = lappend(result, child_joinrel);
+ }
+ return result;
+}
+
+static bool
+is_asymmetric_join_feasible(PlannerInfo *root,
+ RelOptInfo *outer_rel,
+ RelOptInfo *inner_rel,
+ JoinType jointype)
+{
+ ListCell *lc;
+
+ if (jointype != JOIN_INNER && jointype != JOIN_LEFT)
+ return false;
+
+ if (IS_OTHER_REL(outer_rel) || IS_OTHER_REL(inner_rel))
+ return false;
+
+ /* Disallow recursive usage of asymmertic join machinery */
+ if (root->join_rel_level == NULL)
+ return false;
+
+ /*
+ * Don't allow asymmetric JOIN of two append subplans.
+ * In the case of a parameterized NL join, a reparameterization procedure
+ * will lead to large memory allocations and a CPU consumption:
+ * each reparameterization will induce subpath duplication, creating new
+ * ParamPathInfo instance and increasing of ppilist up to number of
+ * partitions in the inner. Also, if we have many partitions, each bitmapset
+ * variable will be large and many leaks of such variable (caused by relid
+ * replacement) will highly increase memory consumption.
+ * So, we deny such paths for now.
+ */
+ foreach(lc, inner_rel->pathlist)
+ {
+ if (IsA(lfirst(lc), AppendPath))
+ return false;
+ }
+
+ return true;
+}
+
+void
+try_asymmetric_partitionwise_join(PlannerInfo *root,
+ RelOptInfo *joinrel,
+ RelOptInfo *outer_rel,
+ RelOptInfo *inner_rel,
+ JoinType jointype,
+ JoinPathExtraData *extra)
+{
+ ListCell *lc;
+
+ /*
+ * Try this kind of paths if we allow complex partitionwise joins and we know
+ * we can build this join safely.
+ */
+ if (!enable_partitionwise_join ||
+ !is_asymmetric_join_feasible(root, outer_rel, inner_rel, jointype))
+ return;
+
+ foreach (lc, outer_rel->pathlist)
+ {
+ AppendPath *append_path = lfirst(lc);
+
+ /*
+ * We assume this pathlist keeps at least one AppendPath that
+ * represents partitioned table-scan, symmetric or asymmetric
+ * partition-wise join. Asymmetric join isn't needed if the append node
+ * has only one child.
+ */
+ if (IsA(append_path, AppendPath) &&
+ list_length(append_path->subpaths) > 1)
+ {
+ List **join_rel_level_saved;
+ List *live_childrels = NIL;
+
+ join_rel_level_saved = root->join_rel_level;
+ PG_TRY();
+ {
+ /* temporary disables "dynamic programming" algorithm */
+ root->join_rel_level = NULL;
+
+ live_childrels =
+ extract_asymmetric_partitionwise_subjoin(root,
+ joinrel,
+ append_path,
+ inner_rel,
+ jointype,
+ extra);
+ }
+ PG_FINALLY();
+ {
+ root->join_rel_level = join_rel_level_saved;
+ }
+ PG_END_TRY();
+
+ if (live_childrels != NIL)
+ add_paths_to_append_rel(root, joinrel, live_childrels);
+
+ break;
+ }
+ }
+}
+
/*
* Construct the SpecialJoinInfo for a child-join by translating
* SpecialJoinInfo for the join between parents. left_relids and right_relids
diff --git a/src/backend/optimizer/plan/setrefs.c b/src/backend/optimizer/plan/setrefs.c
index 61ccfd300b..834843ba33 100644
--- a/src/backend/optimizer/plan/setrefs.c
+++ b/src/backend/optimizer/plan/setrefs.c
@@ -281,24 +281,29 @@ set_plan_references(PlannerInfo *root, Plan *plan)
/*
* Adjust RT indexes of AppendRelInfos and add to final appendrels list.
- * We assume the AppendRelInfos were built during planning and don't need
- * to be copied.
+ * The AppendRelInfos are copied, because as a part of a subplan its could
+ * be visited many times in the case of asymmetric join.
*/
foreach(lc, root->append_rel_list)
{
AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc);
+ AppendRelInfo *newappinfo;
+
+ /* flat copy is enough since all valuable fields are scalars */
+ newappinfo = (AppendRelInfo *) palloc(sizeof(AppendRelInfo));
+ memcpy(newappinfo, appinfo, sizeof(AppendRelInfo));
/* adjust RT indexes */
- appinfo->parent_relid += rtoffset;
- appinfo->child_relid += rtoffset;
+ newappinfo->parent_relid += rtoffset;
+ newappinfo->child_relid += rtoffset;
/*
* Rather than adjust the translated_vars entries, just drop 'em.
* Neither the executor nor EXPLAIN currently need that data.
*/
- appinfo->translated_vars = NIL;
+ newappinfo->translated_vars = NIL;
- glob->appendRelations = lappend(glob->appendRelations, appinfo);
+ glob->appendRelations = lappend(glob->appendRelations, newappinfo);
}
/* Now fix the Plan tree */
diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c
index af46f581ac..ec8dee99b0 100644
--- a/src/backend/optimizer/util/appendinfo.c
+++ b/src/backend/optimizer/util/appendinfo.c
@@ -202,7 +202,9 @@ adjust_appendrel_attrs(PlannerInfo *root, Node *node, int nappinfos,
context.appinfos = appinfos;
/* If there's nothing to adjust, don't call this function. */
- Assert(nappinfos >= 1 && appinfos != NULL);
+ /* If there's nothing to adjust, just return a duplication */
+ if (nappinfos == 0)
+ return copyObject(node);
/* Should never be translating a Query tree. */
Assert(node == NULL || !IsA(node, Query));
@@ -489,12 +491,10 @@ adjust_appendrel_attrs_multilevel(PlannerInfo *root, Node *node,
Relids child_relids,
Relids top_parent_relids)
{
- AppendRelInfo **appinfos;
- Bitmapset *parent_relids = NULL;
- int nappinfos;
- int cnt;
-
- Assert(bms_num_members(child_relids) == bms_num_members(top_parent_relids));
+ AppendRelInfo **appinfos;
+ Relids parent_relids = NULL;
+ int nappinfos;
+ int cnt;
appinfos = find_appinfos_by_relids(root, child_relids, &nappinfos);
@@ -506,8 +506,13 @@ adjust_appendrel_attrs_multilevel(PlannerInfo *root, Node *node,
parent_relids = bms_add_member(parent_relids, appinfo->parent_relid);
}
- /* Recurse if immediate parent is not the top parent. */
- if (!bms_equal(parent_relids, top_parent_relids))
+ /*
+ * Recurse if immediate parent is not the top parent. Keep in mind that in a
+ * case of asymmetric JOIN top_parent_relids can contain relids which aren't
+ * part of an append node.
+ */
+ if (!bms_equal(parent_relids, top_parent_relids) &&
+ !bms_is_subset(parent_relids, top_parent_relids))
node = adjust_appendrel_attrs_multilevel(root, node, parent_relids,
top_parent_relids);
@@ -515,6 +520,7 @@ adjust_appendrel_attrs_multilevel(PlannerInfo *root, Node *node,
node = adjust_appendrel_attrs(root, node, nappinfos, appinfos);
pfree(appinfos);
+ pfree(parent_relids);
return node;
}
@@ -565,6 +571,7 @@ adjust_child_relids_multilevel(PlannerInfo *root, Relids relids,
AppendRelInfo **appinfos;
int nappinfos;
Relids parent_relids = NULL;
+ Relids normal_relids = NULL;
Relids result;
Relids tmp_result = NULL;
int cnt;
@@ -579,12 +586,17 @@ adjust_child_relids_multilevel(PlannerInfo *root, Relids relids,
appinfos = find_appinfos_by_relids(root, child_relids, &nappinfos);
/* Construct relids set for the immediate parent of the given child. */
+ normal_relids = bms_copy(child_relids);
for (cnt = 0; cnt < nappinfos; cnt++)
{
AppendRelInfo *appinfo = appinfos[cnt];
parent_relids = bms_add_member(parent_relids, appinfo->parent_relid);
+ normal_relids = bms_del_member(normal_relids, appinfo->child_relid);
}
+ parent_relids = bms_union(parent_relids, normal_relids);
+ if (normal_relids)
+ bms_free(normal_relids);
/* Recurse if immediate parent is not the top parent. */
if (!bms_equal(parent_relids, top_parent_relids))
@@ -715,11 +727,11 @@ AppendRelInfo **
find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos)
{
AppendRelInfo **appinfos;
+ int nrooms = bms_num_members(relids);
int cnt = 0;
int i;
- *nappinfos = bms_num_members(relids);
- appinfos = (AppendRelInfo **) palloc(sizeof(AppendRelInfo *) * *nappinfos);
+ appinfos = (AppendRelInfo **) palloc(sizeof(AppendRelInfo *) * nrooms);
i = -1;
while ((i = bms_next_member(relids, i)) >= 0)
@@ -727,10 +739,11 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos)
AppendRelInfo *appinfo = root->append_rel_array[i];
if (!appinfo)
- elog(ERROR, "child rel %d not found in append_rel_array", i);
+ continue;
appinfos[cnt++] = appinfo;
}
+ *nappinfos = cnt;
return appinfos;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index 9ce5f95e3b..03c0640473 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -4226,7 +4226,14 @@ do { \
MemoryContextSwitchTo(oldcontext);
}
- bms_free(required_outer);
+
+ /*
+ * If adjust_child_relids_multilevel don't do replacements it returns
+ * the original set, not a copy. It is possible in the case of asymmetric
+ * JOIN and child_rel->relids contains relids only of plane relations.
+ */
+ if (required_outer != old_ppi->ppi_req_outer)
+ bms_free(required_outer);
new_path->param_info = new_ppi;
diff --git a/src/backend/optimizer/util/relnode.c b/src/backend/optimizer/util/relnode.c
index e105a4d5f1..bb15fb2716 100644
--- a/src/backend/optimizer/util/relnode.c
+++ b/src/backend/optimizer/util/relnode.c
@@ -790,11 +790,8 @@ build_child_join_rel(PlannerInfo *root, RelOptInfo *outer_rel,
AppendRelInfo **appinfos;
int nappinfos;
- /* Only joins between "other" relations land here. */
- Assert(IS_OTHER_REL(outer_rel) && IS_OTHER_REL(inner_rel));
-
- /* The parent joinrel should have consider_partitionwise_join set. */
- Assert(parent_joinrel->consider_partitionwise_join);
+ /* Either of relations must be "other" relation at least. */
+ Assert(IS_OTHER_REL(outer_rel) || IS_OTHER_REL(inner_rel));
joinrel->reloptkind = RELOPT_OTHER_JOINREL;
joinrel->relids = bms_union(outer_rel->relids, inner_rel->relids);
@@ -851,8 +848,11 @@ build_child_join_rel(PlannerInfo *root, RelOptInfo *outer_rel,
joinrel->partexprs = NULL;
joinrel->nullable_partexprs = NULL;
- joinrel->top_parent_relids = bms_union(outer_rel->top_parent_relids,
- inner_rel->top_parent_relids);
+ joinrel->top_parent_relids =
+ bms_union(IS_OTHER_REL(outer_rel) ?
+ outer_rel->top_parent_relids : outer_rel->relids,
+ IS_OTHER_REL(inner_rel) ?
+ inner_rel->top_parent_relids : inner_rel->relids);
/* Compute information relevant to foreign relations. */
set_foreign_rel_properties(joinrel, outer_rel, inner_rel);
@@ -2033,9 +2033,8 @@ build_child_join_reltarget(PlannerInfo *root,
{
/* Build the targetlist */
childrel->reltarget->exprs = (List *)
- adjust_appendrel_attrs(root,
- (Node *) parentrel->reltarget->exprs,
- nappinfos, appinfos);
+ adjust_appendrel_attrs_multilevel(root, (Node *)parentrel->reltarget->exprs,
+ childrel->relids, parentrel->relids);
/* Set the cost and width fields */
childrel->reltarget->cost.startup = parentrel->reltarget->cost.startup;
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index f1d111063c..a0106dc539 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -109,7 +109,12 @@ extern bool have_join_order_restriction(PlannerInfo *root,
extern bool have_dangerous_phv(PlannerInfo *root,
Relids outer_relids, Relids inner_params);
extern void mark_dummy_rel(RelOptInfo *rel);
-
+extern void try_asymmetric_partitionwise_join(PlannerInfo *root,
+ RelOptInfo *joinrel,
+ RelOptInfo *outer_rel,
+ RelOptInfo *inner_rel,
+ JoinType jointype,
+ JoinPathExtraData *extra);
/*
* equivclass.c
* routines for managing EquivalenceClasses
diff --git a/src/test/regress/expected/partition_join.out b/src/test/regress/expected/partition_join.out
index 27f7525b3e..601f08662e 100644
--- a/src/test/regress/expected/partition_join.out
+++ b/src/test/regress/expected/partition_join.out
@@ -2320,6 +2320,312 @@ SELECT t1.a, t1.c, t2.b, t2.c FROM prt1_adv t1 INNER JOIN prt2_adv t2 ON (t1.a =
375 | 0375 | 375 | 0375
(8 rows)
+--
+-- For asymmetric partition-wise join
+--
+CREATE TABLE prt5 (hkey int, a int, b int) PARTITION BY HASH(hkey);
+CREATE TABLE prt5_p0 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 0);
+CREATE TABLE prt5_p1 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 1);
+CREATE TABLE prt5_p2 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 2);
+CREATE TABLE prt6 (aid int, alabel text) PARTITION BY HASH(aid);
+CREATE TABLE prt6_p0 PARTITION OF prt6
+ FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE prt6_p1 PARTITION OF prt6
+ FOR VALUES WITH (modulus 2, remainder 1);
+CREATE TABLE t5_1 (aid int, alabel text);
+CREATE TABLE t5_2 (bid int, blabel text);
+INSERT INTO prt5 (SELECT x, (1000.0 * random())::int,
+ (1000.0 * random())::int
+ FROM generate_series(1,1000000) x);
+INSERT INTO t5_1 (SELECT x, md5(x::text) FROM generate_series(-200, 800) x);
+INSERT INTO t5_2 (SELECT x, md5(x::text) FROM generate_series(-200, 800) x);
+INSERT INTO prt6 (SELECT * FROM t5_1);
+VACUUM ANALYZE prt5,prt6,t5_1,t5_2;
+SET max_parallel_workers_per_gather = 0;
+-- Trivial asymmetric JOIN of partitioned table with a relation
+EXPLAIN (COSTS OFF)
+SELECT *
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+-------------------------------------------------------
+ Append
+ -> Hash Join
+ Hash Cond: (prt5_1.a = t5_1.aid)
+ -> Seq Scan on prt5_p0 prt5_1
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_2.a = t5_1.aid)
+ -> Seq Scan on prt5_p1 prt5_2
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_3.a = t5_1.aid)
+ -> Seq Scan on prt5_p2 prt5_3
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+(19 rows)
+
+-- The same, but appended with UNION ALL
+EXPLAIN (COSTS OFF)
+SELECT * FROM (
+ (SELECT * FROM prt5_p0)
+ UNION ALL
+ (SELECT * FROM prt5_p1)
+ UNION ALL
+ (SELECT * FROM prt5_p2)
+ ) AS sq1
+JOIN t5_1 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+-------------------------------------------------------
+ Append
+ -> Hash Join
+ Hash Cond: (prt5_p0.a = t5_1.aid)
+ -> Seq Scan on prt5_p0
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_p1.a = t5_1.aid)
+ -> Seq Scan on prt5_p1
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_p2.a = t5_1.aid)
+ -> Seq Scan on prt5_p2
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+(19 rows)
+
+-- Don't allow asymmetric JOIN of two partitioned tables.
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 JOIN prt6 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+-------------------------------------------------------
+ Hash Join
+ Hash Cond: (prt5.a = prt6.aid)
+ -> Append
+ -> Seq Scan on prt5_p0 prt5_1
+ -> Seq Scan on prt5_p1 prt5_2
+ -> Seq Scan on prt5_p2 prt5_3
+ -> Hash
+ -> Append
+ -> Seq Scan on prt6_p0 prt6_1
+ Filter: (alabel ~~ '%abc%'::text)
+ -> Seq Scan on prt6_p1 prt6_2
+ Filter: (alabel ~~ '%abc%'::text)
+(12 rows)
+
+-- Check asymmetric JOIN with Subquery
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 JOIN (
+ SELECT * FROM prt6 LIMIT 1000
+) AS sq1 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+----------------------------------------------------------------
+ Append
+ -> Hash Join
+ Hash Cond: (prt5_1.a = sq1.aid)
+ -> Seq Scan on prt5_p0 prt5_1
+ -> Hash
+ -> Subquery Scan on sq1
+ Filter: (sq1.alabel ~~ '%abc%'::text)
+ -> Limit
+ -> Append
+ -> Seq Scan on prt6_p0 prt6_1
+ -> Seq Scan on prt6_p1 prt6_2
+ -> Hash Join
+ Hash Cond: (prt5_2.a = sq1.aid)
+ -> Seq Scan on prt5_p1 prt5_2
+ -> Hash
+ -> Subquery Scan on sq1
+ Filter: (sq1.alabel ~~ '%abc%'::text)
+ -> Limit
+ -> Append
+ -> Seq Scan on prt6_p0 prt6_4
+ -> Seq Scan on prt6_p1 prt6_5
+ -> Hash Join
+ Hash Cond: (prt5_3.a = sq1.aid)
+ -> Seq Scan on prt5_p2 prt5_3
+ -> Hash
+ -> Subquery Scan on sq1
+ Filter: (sq1.alabel ~~ '%abc%'::text)
+ -> Limit
+ -> Append
+ -> Seq Scan on prt6_p0 prt6_7
+ -> Seq Scan on prt6_p1 prt6_8
+(31 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT *
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+ QUERY PLAN
+------------------------------------------------------------
+ Append
+ -> Hash Join
+ Hash Cond: (prt5_1.a = t5_1.aid)
+ -> Hash Join
+ Hash Cond: (prt5_1.b = t5_2.bid)
+ -> Seq Scan on prt5_p0 prt5_1
+ -> Hash
+ -> Seq Scan on t5_2
+ Filter: (blabel ~~ '%cd%'::text)
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%ab%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_2.a = t5_1.aid)
+ -> Hash Join
+ Hash Cond: (prt5_2.b = t5_2.bid)
+ -> Seq Scan on prt5_p1 prt5_2
+ -> Hash
+ -> Seq Scan on t5_2
+ Filter: (blabel ~~ '%cd%'::text)
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%ab%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_3.a = t5_1.aid)
+ -> Hash Join
+ Hash Cond: (prt5_3.b = t5_2.bid)
+ -> Seq Scan on prt5_p2 prt5_3
+ -> Hash
+ -> Seq Scan on t5_2
+ Filter: (blabel ~~ '%cd%'::text)
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%ab%'::text)
+(34 rows)
+
+-- unable to extract non-partitioned right relation
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 RIGHT JOIN t5_1 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+-----------------------------------------------
+ Hash Right Join
+ Hash Cond: (prt5.a = t5_1.aid)
+ Join Filter: (t5_1.alabel ~~ '%abc%'::text)
+ -> Append
+ -> Seq Scan on prt5_p0 prt5_1
+ -> Seq Scan on prt5_p1 prt5_2
+ -> Seq Scan on prt5_p2 prt5_3
+ -> Hash
+ -> Seq Scan on t5_1
+(9 rows)
+
+-- left side can be extracted, but no cost benefit
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 LEFT JOIN t5_1 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+-------------------------------------------------
+ Hash Left Join
+ Hash Cond: (prt5.a = t5_1.aid)
+ -> Append
+ -> Seq Scan on prt5_p0 prt5_1
+ -> Seq Scan on prt5_p1 prt5_2
+ -> Seq Scan on prt5_p2 prt5_3
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+(9 rows)
+
+-- validation of the results with/without asymmetric partition-wise join
+SELECT * INTO pg_temp.result01a
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+SELECT * INTO pg_temp.result02a
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+SET enable_partitionwise_join = off;
+SELECT * INTO pg_temp.result01b
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+SELECT * INTO pg_temp.result02b
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+SELECT * FROM pg_temp.result01a EXCEPT SELECT * FROM pg_temp.result01b;
+ hkey | a | b | aid | alabel
+------+---+---+-----+--------
+(0 rows)
+
+SELECT * FROM pg_temp.result02a EXCEPT SELECT * FROM pg_temp.result02b;
+ hkey | a | b | aid | alabel | bid | blabel
+------+---+---+-----+--------+-----+--------
+(0 rows)
+
+RESET max_parallel_workers_per_gather;
+SET enable_partitionwise_join = true;
+-- Parameterized path examples.
+CREATE TABLE prta (id integer, payload text) PARTITION BY HASH (id);
+CREATE TABLE prta1 PARTITION OF prta FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE prta2 PARTITION OF prta FOR VALUES WITH (modulus 2, remainder 1);
+CREATE UNIQUE INDEX ON prta1 (id);
+CREATE UNIQUE INDEX ON prta2 (id);
+INSERT INTO prta (id, payload)
+ (SELECT *, ('abc' || id)::text AS payload
+ FROM generate_series(1,1) AS id);
+CREATE TABLE prtb (id integer, payload text) PARTITION BY HASH (id);
+CREATE TABLE prtb1 PARTITION OF prtb FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE prtb2 PARTITION OF prtb FOR VALUES WITH (modulus 2, remainder 1);
+CREATE UNIQUE INDEX ON prtb1 (id);
+CREATE UNIQUE INDEX ON prtb2 (id);
+INSERT INTO prtb (id, payload)
+ (SELECT *, ('def' || id)::text AS payload
+ FROM generate_series(1,1000) AS id);
+CREATE TABLE e (id integer, payload text) PARTITION BY HASH (id);
+CREATE TABLE e1 PARTITION OF e FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE e2 PARTITION OF e FOR VALUES WITH (modulus 2, remainder 1);
+INSERT INTO e (id, payload)
+ (SELECT *, ('ghi' || id)::text AS payload
+ FROM generate_series(1,1000) AS id);
+CREATE UNIQUE INDEX ON e1 (id);
+CREATE UNIQUE INDEX ON e2 (id);
+ANALYZE prta,prtb,e;
+EXPLAIN (COSTS OFF)
+SELECT * FROM prta,prtb WHERE prta.id=prtb.id;
+ QUERY PLAN
+-----------------------------------------------------------
+ Append
+ -> Nested Loop
+ -> Seq Scan on prta1 prta_1
+ -> Index Scan using prtb1_id_idx on prtb1 prtb_1
+ Index Cond: (id = prta_1.id)
+ -> Nested Loop
+ -> Seq Scan on prta2 prta_2
+ -> Index Scan using prtb2_id_idx on prtb2 prtb_2
+ Index Cond: (id = prta_2.id)
+(9 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT * FROM prta,prtb,e WHERE prta.id=prtb.id AND prta.id=e.id;
+ QUERY PLAN
+-----------------------------------------------------------------
+ Append
+ -> Nested Loop
+ Join Filter: (prta_1.id = e_1.id)
+ -> Nested Loop
+ -> Seq Scan on prta1 prta_1
+ -> Index Scan using prtb1_id_idx on prtb1 prtb_1
+ Index Cond: (id = prta_1.id)
+ -> Index Scan using e1_id_idx on e1 e_1
+ Index Cond: (id = prtb_1.id)
+ -> Nested Loop
+ Join Filter: (prta_2.id = e_2.id)
+ -> Nested Loop
+ -> Seq Scan on prta2 prta_2
+ -> Index Scan using prtb2_id_idx on prtb2 prtb_2
+ Index Cond: (id = prta_2.id)
+ -> Index Scan using e2_id_idx on e2 e_2
+ Index Cond: (id = prtb_2.id)
+(17 rows)
+
-- semi join
EXPLAIN (COSTS OFF)
SELECT t1.* FROM prt1_adv t1 WHERE EXISTS (SELECT 1 FROM prt2_adv t2 WHERE t1.a = t2.b) AND t1.b = 0 ORDER BY t1.a;
diff --git a/src/test/regress/sql/partition_join.sql b/src/test/regress/sql/partition_join.sql
index d97b5b69ff..51b81ce3f8 100644
--- a/src/test/regress/sql/partition_join.sql
+++ b/src/test/regress/sql/partition_join.sql
@@ -536,6 +536,132 @@ EXPLAIN (COSTS OFF)
SELECT t1.a, t1.c, t2.b, t2.c FROM prt1_adv t1 INNER JOIN prt2_adv t2 ON (t1.a = t2.b) WHERE t1.b = 0 ORDER BY t1.a, t2.b;
SELECT t1.a, t1.c, t2.b, t2.c FROM prt1_adv t1 INNER JOIN prt2_adv t2 ON (t1.a = t2.b) WHERE t1.b = 0 ORDER BY t1.a, t2.b;
+--
+-- For asymmetric partition-wise join
+--
+CREATE TABLE prt5 (hkey int, a int, b int) PARTITION BY HASH(hkey);
+CREATE TABLE prt5_p0 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 0);
+CREATE TABLE prt5_p1 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 1);
+CREATE TABLE prt5_p2 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 2);
+CREATE TABLE prt6 (aid int, alabel text) PARTITION BY HASH(aid);
+CREATE TABLE prt6_p0 PARTITION OF prt6
+ FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE prt6_p1 PARTITION OF prt6
+ FOR VALUES WITH (modulus 2, remainder 1);
+CREATE TABLE t5_1 (aid int, alabel text);
+CREATE TABLE t5_2 (bid int, blabel text);
+
+INSERT INTO prt5 (SELECT x, (1000.0 * random())::int,
+ (1000.0 * random())::int
+ FROM generate_series(1,1000000) x);
+INSERT INTO t5_1 (SELECT x, md5(x::text) FROM generate_series(-200, 800) x);
+INSERT INTO t5_2 (SELECT x, md5(x::text) FROM generate_series(-200, 800) x);
+INSERT INTO prt6 (SELECT * FROM t5_1);
+
+VACUUM ANALYZE prt5,prt6,t5_1,t5_2;
+
+SET max_parallel_workers_per_gather = 0;
+
+-- Trivial asymmetric JOIN of partitioned table with a relation
+EXPLAIN (COSTS OFF)
+SELECT *
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+
+-- The same, but appended with UNION ALL
+EXPLAIN (COSTS OFF)
+SELECT * FROM (
+ (SELECT * FROM prt5_p0)
+ UNION ALL
+ (SELECT * FROM prt5_p1)
+ UNION ALL
+ (SELECT * FROM prt5_p2)
+ ) AS sq1
+JOIN t5_1 ON a = aid AND alabel like '%abc%';
+
+-- Don't allow asymmetric JOIN of two partitioned tables.
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 JOIN prt6 ON a = aid AND alabel like '%abc%';
+
+-- Check asymmetric JOIN with Subquery
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 JOIN (
+ SELECT * FROM prt6 LIMIT 1000
+) AS sq1 ON a = aid AND alabel like '%abc%';
+
+EXPLAIN (COSTS OFF)
+SELECT *
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+
+-- unable to extract non-partitioned right relation
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 RIGHT JOIN t5_1 ON a = aid AND alabel like '%abc%';
+-- left side can be extracted, but no cost benefit
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 LEFT JOIN t5_1 ON a = aid AND alabel like '%abc%';
+
+-- validation of the results with/without asymmetric partition-wise join
+SELECT * INTO pg_temp.result01a
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+
+SELECT * INTO pg_temp.result02a
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+
+SET enable_partitionwise_join = off;
+
+SELECT * INTO pg_temp.result01b
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+
+SELECT * INTO pg_temp.result02b
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+
+SELECT * FROM pg_temp.result01a EXCEPT SELECT * FROM pg_temp.result01b;
+SELECT * FROM pg_temp.result02a EXCEPT SELECT * FROM pg_temp.result02b;
+
+RESET max_parallel_workers_per_gather;
+SET enable_partitionwise_join = true;
+
+-- Parameterized path examples.
+CREATE TABLE prta (id integer, payload text) PARTITION BY HASH (id);
+CREATE TABLE prta1 PARTITION OF prta FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE prta2 PARTITION OF prta FOR VALUES WITH (modulus 2, remainder 1);
+CREATE UNIQUE INDEX ON prta1 (id);
+CREATE UNIQUE INDEX ON prta2 (id);
+INSERT INTO prta (id, payload)
+ (SELECT *, ('abc' || id)::text AS payload
+ FROM generate_series(1,1) AS id);
+
+CREATE TABLE prtb (id integer, payload text) PARTITION BY HASH (id);
+CREATE TABLE prtb1 PARTITION OF prtb FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE prtb2 PARTITION OF prtb FOR VALUES WITH (modulus 2, remainder 1);
+CREATE UNIQUE INDEX ON prtb1 (id);
+CREATE UNIQUE INDEX ON prtb2 (id);
+INSERT INTO prtb (id, payload)
+ (SELECT *, ('def' || id)::text AS payload
+ FROM generate_series(1,1000) AS id);
+
+CREATE TABLE e (id integer, payload text) PARTITION BY HASH (id);
+CREATE TABLE e1 PARTITION OF e FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE e2 PARTITION OF e FOR VALUES WITH (modulus 2, remainder 1);
+INSERT INTO e (id, payload)
+ (SELECT *, ('ghi' || id)::text AS payload
+ FROM generate_series(1,1000) AS id);
+CREATE UNIQUE INDEX ON e1 (id);
+CREATE UNIQUE INDEX ON e2 (id);
+
+ANALYZE prta,prtb,e;
+
+EXPLAIN (COSTS OFF)
+SELECT * FROM prta,prtb WHERE prta.id=prtb.id;
+
+EXPLAIN (COSTS OFF)
+SELECT * FROM prta,prtb,e WHERE prta.id=prtb.id AND prta.id=e.id;
+
-- semi join
EXPLAIN (COSTS OFF)
SELECT t1.* FROM prt1_adv t1 WHERE EXISTS (SELECT 1 FROM prt2_adv t2 WHERE t1.a = t2.b) AND t1.b = 0 ORDER BY t1.a;
--
2.31.1
Attachments:
[text/plain] v16-0001-Asymmetric-partitionwise-join.patch (33.0K, ../../[email protected]/2-v16-0001-Asymmetric-partitionwise-join.patch)
download | inline diff:
From b9d7a148ab51f297c98345127e975f864fe6ef92 Mon Sep 17 00:00:00 2001
From: Andrey Lepikhov <[email protected]>
Date: Fri, 2 Apr 2021 11:02:20 +0500
Subject: [PATCH] Asymmetric partitionwise join.
Teach optimizer to consider partitionwise join of non-partitioned
table with each partition of partitioned table.
Disallow asymmetric machinery for joining of two partitioned (or appended)
relations because it could cause huge consumption of CPU and memory
during reparameterization of NestLoop path.
---
src/backend/optimizer/path/joinpath.c | 9 +
src/backend/optimizer/path/joinrels.c | 172 +++++++++++
src/backend/optimizer/plan/setrefs.c | 17 +-
src/backend/optimizer/util/appendinfo.c | 37 ++-
src/backend/optimizer/util/pathnode.c | 9 +-
src/backend/optimizer/util/relnode.c | 19 +-
src/include/optimizer/paths.h | 7 +-
src/test/regress/expected/partition_join.out | 306 +++++++++++++++++++
src/test/regress/sql/partition_join.sql | 126 ++++++++
9 files changed, 672 insertions(+), 30 deletions(-)
diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c
index b67b517770..7a1a7b2e86 100644
--- a/src/backend/optimizer/path/joinpath.c
+++ b/src/backend/optimizer/path/joinpath.c
@@ -335,6 +335,15 @@ add_paths_to_joinrel(PlannerInfo *root,
if (set_join_pathlist_hook)
set_join_pathlist_hook(root, joinrel, outerrel, innerrel,
jointype, &extra);
+
+ /*
+ * 7. If outer relation is delivered from partition-tables, consider
+ * distributing inner relation to every partition-leaf prior to
+ * append these leafs.
+ */
+ try_asymmetric_partitionwise_join(root, joinrel,
+ outerrel, innerrel,
+ jointype, &extra);
}
/*
diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c
index 0dbe2ac726..e9cbff9709 100644
--- a/src/backend/optimizer/path/joinrels.c
+++ b/src/backend/optimizer/path/joinrels.c
@@ -16,6 +16,7 @@
#include "miscadmin.h"
#include "optimizer/appendinfo.h"
+#include "optimizer/cost.h"
#include "optimizer/joininfo.h"
#include "optimizer/pathnode.h"
#include "optimizer/paths.h"
@@ -1551,6 +1552,177 @@ try_partitionwise_join(PlannerInfo *root, RelOptInfo *rel1, RelOptInfo *rel2,
}
}
+/*
+ * Build RelOptInfo on JOIN of each partition of the outer relation and the inner
+ * relation. Return List of such RelOptInfo's. Return NIL, if at least one of
+ * these JOINs is impossible to build.
+ */
+static List *
+extract_asymmetric_partitionwise_subjoin(PlannerInfo *root,
+ RelOptInfo *joinrel,
+ AppendPath *append_path,
+ RelOptInfo *inner_rel,
+ JoinType jointype,
+ JoinPathExtraData *extra)
+{
+ List *result = NIL;
+ ListCell *lc;
+
+ foreach (lc, append_path->subpaths)
+ {
+ Path *child_path = lfirst(lc);
+ RelOptInfo *child_rel = child_path->parent;
+ Relids child_joinrelids;
+ Relids parent_relids;
+ RelOptInfo *child_joinrel;
+ SpecialJoinInfo *child_sjinfo;
+ List *child_restrictlist;
+
+ child_joinrelids = bms_union(child_rel->relids, inner_rel->relids);
+ parent_relids = bms_union(append_path->path.parent->relids,
+ inner_rel->relids);
+
+ child_sjinfo = build_child_join_sjinfo(root, extra->sjinfo,
+ child_rel->relids,
+ inner_rel->relids);
+ child_restrictlist = (List *)
+ adjust_appendrel_attrs_multilevel(root, (Node *)extra->restrictlist,
+ child_joinrelids, parent_relids);
+
+ child_joinrel = find_join_rel(root, child_joinrelids);
+ if (!child_joinrel)
+ child_joinrel = build_child_join_rel(root,
+ child_rel,
+ inner_rel,
+ joinrel,
+ child_restrictlist,
+ child_sjinfo,
+ jointype);
+ else
+ {
+ /*
+ * The join relation already exists. For example, it could happen if
+ * we join two plane tables with partitioned table(s). Do nothing.
+ */
+ }
+
+ populate_joinrel_with_paths(root,
+ child_rel,
+ inner_rel,
+ child_joinrel,
+ child_sjinfo,
+ child_restrictlist);
+
+ /* Give up if asymmetric partition-wise join is not available */
+ if (child_joinrel->pathlist == NIL)
+ return NIL;
+
+ set_cheapest(child_joinrel);
+ result = lappend(result, child_joinrel);
+ }
+ return result;
+}
+
+static bool
+is_asymmetric_join_feasible(PlannerInfo *root,
+ RelOptInfo *outer_rel,
+ RelOptInfo *inner_rel,
+ JoinType jointype)
+{
+ ListCell *lc;
+
+ if (jointype != JOIN_INNER && jointype != JOIN_LEFT)
+ return false;
+
+ if (IS_OTHER_REL(outer_rel) || IS_OTHER_REL(inner_rel))
+ return false;
+
+ /* Disallow recursive usage of asymmertic join machinery */
+ if (root->join_rel_level == NULL)
+ return false;
+
+ /*
+ * Don't allow asymmetric JOIN of two append subplans.
+ * In the case of a parameterized NL join, a reparameterization procedure
+ * will lead to large memory allocations and a CPU consumption:
+ * each reparameterization will induce subpath duplication, creating new
+ * ParamPathInfo instance and increasing of ppilist up to number of
+ * partitions in the inner. Also, if we have many partitions, each bitmapset
+ * variable will be large and many leaks of such variable (caused by relid
+ * replacement) will highly increase memory consumption.
+ * So, we deny such paths for now.
+ */
+ foreach(lc, inner_rel->pathlist)
+ {
+ if (IsA(lfirst(lc), AppendPath))
+ return false;
+ }
+
+ return true;
+}
+
+void
+try_asymmetric_partitionwise_join(PlannerInfo *root,
+ RelOptInfo *joinrel,
+ RelOptInfo *outer_rel,
+ RelOptInfo *inner_rel,
+ JoinType jointype,
+ JoinPathExtraData *extra)
+{
+ ListCell *lc;
+
+ /*
+ * Try this kind of paths if we allow complex partitionwise joins and we know
+ * we can build this join safely.
+ */
+ if (!enable_partitionwise_join ||
+ !is_asymmetric_join_feasible(root, outer_rel, inner_rel, jointype))
+ return;
+
+ foreach (lc, outer_rel->pathlist)
+ {
+ AppendPath *append_path = lfirst(lc);
+
+ /*
+ * We assume this pathlist keeps at least one AppendPath that
+ * represents partitioned table-scan, symmetric or asymmetric
+ * partition-wise join. Asymmetric join isn't needed if the append node
+ * has only one child.
+ */
+ if (IsA(append_path, AppendPath) &&
+ list_length(append_path->subpaths) > 1)
+ {
+ List **join_rel_level_saved;
+ List *live_childrels = NIL;
+
+ join_rel_level_saved = root->join_rel_level;
+ PG_TRY();
+ {
+ /* temporary disables "dynamic programming" algorithm */
+ root->join_rel_level = NULL;
+
+ live_childrels =
+ extract_asymmetric_partitionwise_subjoin(root,
+ joinrel,
+ append_path,
+ inner_rel,
+ jointype,
+ extra);
+ }
+ PG_FINALLY();
+ {
+ root->join_rel_level = join_rel_level_saved;
+ }
+ PG_END_TRY();
+
+ if (live_childrels != NIL)
+ add_paths_to_append_rel(root, joinrel, live_childrels);
+
+ break;
+ }
+ }
+}
+
/*
* Construct the SpecialJoinInfo for a child-join by translating
* SpecialJoinInfo for the join between parents. left_relids and right_relids
diff --git a/src/backend/optimizer/plan/setrefs.c b/src/backend/optimizer/plan/setrefs.c
index 61ccfd300b..834843ba33 100644
--- a/src/backend/optimizer/plan/setrefs.c
+++ b/src/backend/optimizer/plan/setrefs.c
@@ -281,24 +281,29 @@ set_plan_references(PlannerInfo *root, Plan *plan)
/*
* Adjust RT indexes of AppendRelInfos and add to final appendrels list.
- * We assume the AppendRelInfos were built during planning and don't need
- * to be copied.
+ * The AppendRelInfos are copied, because as a part of a subplan its could
+ * be visited many times in the case of asymmetric join.
*/
foreach(lc, root->append_rel_list)
{
AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc);
+ AppendRelInfo *newappinfo;
+
+ /* flat copy is enough since all valuable fields are scalars */
+ newappinfo = (AppendRelInfo *) palloc(sizeof(AppendRelInfo));
+ memcpy(newappinfo, appinfo, sizeof(AppendRelInfo));
/* adjust RT indexes */
- appinfo->parent_relid += rtoffset;
- appinfo->child_relid += rtoffset;
+ newappinfo->parent_relid += rtoffset;
+ newappinfo->child_relid += rtoffset;
/*
* Rather than adjust the translated_vars entries, just drop 'em.
* Neither the executor nor EXPLAIN currently need that data.
*/
- appinfo->translated_vars = NIL;
+ newappinfo->translated_vars = NIL;
- glob->appendRelations = lappend(glob->appendRelations, appinfo);
+ glob->appendRelations = lappend(glob->appendRelations, newappinfo);
}
/* Now fix the Plan tree */
diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c
index af46f581ac..ec8dee99b0 100644
--- a/src/backend/optimizer/util/appendinfo.c
+++ b/src/backend/optimizer/util/appendinfo.c
@@ -202,7 +202,9 @@ adjust_appendrel_attrs(PlannerInfo *root, Node *node, int nappinfos,
context.appinfos = appinfos;
/* If there's nothing to adjust, don't call this function. */
- Assert(nappinfos >= 1 && appinfos != NULL);
+ /* If there's nothing to adjust, just return a duplication */
+ if (nappinfos == 0)
+ return copyObject(node);
/* Should never be translating a Query tree. */
Assert(node == NULL || !IsA(node, Query));
@@ -489,12 +491,10 @@ adjust_appendrel_attrs_multilevel(PlannerInfo *root, Node *node,
Relids child_relids,
Relids top_parent_relids)
{
- AppendRelInfo **appinfos;
- Bitmapset *parent_relids = NULL;
- int nappinfos;
- int cnt;
-
- Assert(bms_num_members(child_relids) == bms_num_members(top_parent_relids));
+ AppendRelInfo **appinfos;
+ Relids parent_relids = NULL;
+ int nappinfos;
+ int cnt;
appinfos = find_appinfos_by_relids(root, child_relids, &nappinfos);
@@ -506,8 +506,13 @@ adjust_appendrel_attrs_multilevel(PlannerInfo *root, Node *node,
parent_relids = bms_add_member(parent_relids, appinfo->parent_relid);
}
- /* Recurse if immediate parent is not the top parent. */
- if (!bms_equal(parent_relids, top_parent_relids))
+ /*
+ * Recurse if immediate parent is not the top parent. Keep in mind that in a
+ * case of asymmetric JOIN top_parent_relids can contain relids which aren't
+ * part of an append node.
+ */
+ if (!bms_equal(parent_relids, top_parent_relids) &&
+ !bms_is_subset(parent_relids, top_parent_relids))
node = adjust_appendrel_attrs_multilevel(root, node, parent_relids,
top_parent_relids);
@@ -515,6 +520,7 @@ adjust_appendrel_attrs_multilevel(PlannerInfo *root, Node *node,
node = adjust_appendrel_attrs(root, node, nappinfos, appinfos);
pfree(appinfos);
+ pfree(parent_relids);
return node;
}
@@ -565,6 +571,7 @@ adjust_child_relids_multilevel(PlannerInfo *root, Relids relids,
AppendRelInfo **appinfos;
int nappinfos;
Relids parent_relids = NULL;
+ Relids normal_relids = NULL;
Relids result;
Relids tmp_result = NULL;
int cnt;
@@ -579,12 +586,17 @@ adjust_child_relids_multilevel(PlannerInfo *root, Relids relids,
appinfos = find_appinfos_by_relids(root, child_relids, &nappinfos);
/* Construct relids set for the immediate parent of the given child. */
+ normal_relids = bms_copy(child_relids);
for (cnt = 0; cnt < nappinfos; cnt++)
{
AppendRelInfo *appinfo = appinfos[cnt];
parent_relids = bms_add_member(parent_relids, appinfo->parent_relid);
+ normal_relids = bms_del_member(normal_relids, appinfo->child_relid);
}
+ parent_relids = bms_union(parent_relids, normal_relids);
+ if (normal_relids)
+ bms_free(normal_relids);
/* Recurse if immediate parent is not the top parent. */
if (!bms_equal(parent_relids, top_parent_relids))
@@ -715,11 +727,11 @@ AppendRelInfo **
find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos)
{
AppendRelInfo **appinfos;
+ int nrooms = bms_num_members(relids);
int cnt = 0;
int i;
- *nappinfos = bms_num_members(relids);
- appinfos = (AppendRelInfo **) palloc(sizeof(AppendRelInfo *) * *nappinfos);
+ appinfos = (AppendRelInfo **) palloc(sizeof(AppendRelInfo *) * nrooms);
i = -1;
while ((i = bms_next_member(relids, i)) >= 0)
@@ -727,10 +739,11 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos)
AppendRelInfo *appinfo = root->append_rel_array[i];
if (!appinfo)
- elog(ERROR, "child rel %d not found in append_rel_array", i);
+ continue;
appinfos[cnt++] = appinfo;
}
+ *nappinfos = cnt;
return appinfos;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index 9ce5f95e3b..03c0640473 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -4226,7 +4226,14 @@ do { \
MemoryContextSwitchTo(oldcontext);
}
- bms_free(required_outer);
+
+ /*
+ * If adjust_child_relids_multilevel don't do replacements it returns
+ * the original set, not a copy. It is possible in the case of asymmetric
+ * JOIN and child_rel->relids contains relids only of plane relations.
+ */
+ if (required_outer != old_ppi->ppi_req_outer)
+ bms_free(required_outer);
new_path->param_info = new_ppi;
diff --git a/src/backend/optimizer/util/relnode.c b/src/backend/optimizer/util/relnode.c
index e105a4d5f1..bb15fb2716 100644
--- a/src/backend/optimizer/util/relnode.c
+++ b/src/backend/optimizer/util/relnode.c
@@ -790,11 +790,8 @@ build_child_join_rel(PlannerInfo *root, RelOptInfo *outer_rel,
AppendRelInfo **appinfos;
int nappinfos;
- /* Only joins between "other" relations land here. */
- Assert(IS_OTHER_REL(outer_rel) && IS_OTHER_REL(inner_rel));
-
- /* The parent joinrel should have consider_partitionwise_join set. */
- Assert(parent_joinrel->consider_partitionwise_join);
+ /* Either of relations must be "other" relation at least. */
+ Assert(IS_OTHER_REL(outer_rel) || IS_OTHER_REL(inner_rel));
joinrel->reloptkind = RELOPT_OTHER_JOINREL;
joinrel->relids = bms_union(outer_rel->relids, inner_rel->relids);
@@ -851,8 +848,11 @@ build_child_join_rel(PlannerInfo *root, RelOptInfo *outer_rel,
joinrel->partexprs = NULL;
joinrel->nullable_partexprs = NULL;
- joinrel->top_parent_relids = bms_union(outer_rel->top_parent_relids,
- inner_rel->top_parent_relids);
+ joinrel->top_parent_relids =
+ bms_union(IS_OTHER_REL(outer_rel) ?
+ outer_rel->top_parent_relids : outer_rel->relids,
+ IS_OTHER_REL(inner_rel) ?
+ inner_rel->top_parent_relids : inner_rel->relids);
/* Compute information relevant to foreign relations. */
set_foreign_rel_properties(joinrel, outer_rel, inner_rel);
@@ -2033,9 +2033,8 @@ build_child_join_reltarget(PlannerInfo *root,
{
/* Build the targetlist */
childrel->reltarget->exprs = (List *)
- adjust_appendrel_attrs(root,
- (Node *) parentrel->reltarget->exprs,
- nappinfos, appinfos);
+ adjust_appendrel_attrs_multilevel(root, (Node *)parentrel->reltarget->exprs,
+ childrel->relids, parentrel->relids);
/* Set the cost and width fields */
childrel->reltarget->cost.startup = parentrel->reltarget->cost.startup;
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index f1d111063c..a0106dc539 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -109,7 +109,12 @@ extern bool have_join_order_restriction(PlannerInfo *root,
extern bool have_dangerous_phv(PlannerInfo *root,
Relids outer_relids, Relids inner_params);
extern void mark_dummy_rel(RelOptInfo *rel);
-
+extern void try_asymmetric_partitionwise_join(PlannerInfo *root,
+ RelOptInfo *joinrel,
+ RelOptInfo *outer_rel,
+ RelOptInfo *inner_rel,
+ JoinType jointype,
+ JoinPathExtraData *extra);
/*
* equivclass.c
* routines for managing EquivalenceClasses
diff --git a/src/test/regress/expected/partition_join.out b/src/test/regress/expected/partition_join.out
index 27f7525b3e..601f08662e 100644
--- a/src/test/regress/expected/partition_join.out
+++ b/src/test/regress/expected/partition_join.out
@@ -2320,6 +2320,312 @@ SELECT t1.a, t1.c, t2.b, t2.c FROM prt1_adv t1 INNER JOIN prt2_adv t2 ON (t1.a =
375 | 0375 | 375 | 0375
(8 rows)
+--
+-- For asymmetric partition-wise join
+--
+CREATE TABLE prt5 (hkey int, a int, b int) PARTITION BY HASH(hkey);
+CREATE TABLE prt5_p0 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 0);
+CREATE TABLE prt5_p1 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 1);
+CREATE TABLE prt5_p2 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 2);
+CREATE TABLE prt6 (aid int, alabel text) PARTITION BY HASH(aid);
+CREATE TABLE prt6_p0 PARTITION OF prt6
+ FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE prt6_p1 PARTITION OF prt6
+ FOR VALUES WITH (modulus 2, remainder 1);
+CREATE TABLE t5_1 (aid int, alabel text);
+CREATE TABLE t5_2 (bid int, blabel text);
+INSERT INTO prt5 (SELECT x, (1000.0 * random())::int,
+ (1000.0 * random())::int
+ FROM generate_series(1,1000000) x);
+INSERT INTO t5_1 (SELECT x, md5(x::text) FROM generate_series(-200, 800) x);
+INSERT INTO t5_2 (SELECT x, md5(x::text) FROM generate_series(-200, 800) x);
+INSERT INTO prt6 (SELECT * FROM t5_1);
+VACUUM ANALYZE prt5,prt6,t5_1,t5_2;
+SET max_parallel_workers_per_gather = 0;
+-- Trivial asymmetric JOIN of partitioned table with a relation
+EXPLAIN (COSTS OFF)
+SELECT *
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+-------------------------------------------------------
+ Append
+ -> Hash Join
+ Hash Cond: (prt5_1.a = t5_1.aid)
+ -> Seq Scan on prt5_p0 prt5_1
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_2.a = t5_1.aid)
+ -> Seq Scan on prt5_p1 prt5_2
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_3.a = t5_1.aid)
+ -> Seq Scan on prt5_p2 prt5_3
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+(19 rows)
+
+-- The same, but appended with UNION ALL
+EXPLAIN (COSTS OFF)
+SELECT * FROM (
+ (SELECT * FROM prt5_p0)
+ UNION ALL
+ (SELECT * FROM prt5_p1)
+ UNION ALL
+ (SELECT * FROM prt5_p2)
+ ) AS sq1
+JOIN t5_1 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+-------------------------------------------------------
+ Append
+ -> Hash Join
+ Hash Cond: (prt5_p0.a = t5_1.aid)
+ -> Seq Scan on prt5_p0
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_p1.a = t5_1.aid)
+ -> Seq Scan on prt5_p1
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_p2.a = t5_1.aid)
+ -> Seq Scan on prt5_p2
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+(19 rows)
+
+-- Don't allow asymmetric JOIN of two partitioned tables.
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 JOIN prt6 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+-------------------------------------------------------
+ Hash Join
+ Hash Cond: (prt5.a = prt6.aid)
+ -> Append
+ -> Seq Scan on prt5_p0 prt5_1
+ -> Seq Scan on prt5_p1 prt5_2
+ -> Seq Scan on prt5_p2 prt5_3
+ -> Hash
+ -> Append
+ -> Seq Scan on prt6_p0 prt6_1
+ Filter: (alabel ~~ '%abc%'::text)
+ -> Seq Scan on prt6_p1 prt6_2
+ Filter: (alabel ~~ '%abc%'::text)
+(12 rows)
+
+-- Check asymmetric JOIN with Subquery
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 JOIN (
+ SELECT * FROM prt6 LIMIT 1000
+) AS sq1 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+----------------------------------------------------------------
+ Append
+ -> Hash Join
+ Hash Cond: (prt5_1.a = sq1.aid)
+ -> Seq Scan on prt5_p0 prt5_1
+ -> Hash
+ -> Subquery Scan on sq1
+ Filter: (sq1.alabel ~~ '%abc%'::text)
+ -> Limit
+ -> Append
+ -> Seq Scan on prt6_p0 prt6_1
+ -> Seq Scan on prt6_p1 prt6_2
+ -> Hash Join
+ Hash Cond: (prt5_2.a = sq1.aid)
+ -> Seq Scan on prt5_p1 prt5_2
+ -> Hash
+ -> Subquery Scan on sq1
+ Filter: (sq1.alabel ~~ '%abc%'::text)
+ -> Limit
+ -> Append
+ -> Seq Scan on prt6_p0 prt6_4
+ -> Seq Scan on prt6_p1 prt6_5
+ -> Hash Join
+ Hash Cond: (prt5_3.a = sq1.aid)
+ -> Seq Scan on prt5_p2 prt5_3
+ -> Hash
+ -> Subquery Scan on sq1
+ Filter: (sq1.alabel ~~ '%abc%'::text)
+ -> Limit
+ -> Append
+ -> Seq Scan on prt6_p0 prt6_7
+ -> Seq Scan on prt6_p1 prt6_8
+(31 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT *
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+ QUERY PLAN
+------------------------------------------------------------
+ Append
+ -> Hash Join
+ Hash Cond: (prt5_1.a = t5_1.aid)
+ -> Hash Join
+ Hash Cond: (prt5_1.b = t5_2.bid)
+ -> Seq Scan on prt5_p0 prt5_1
+ -> Hash
+ -> Seq Scan on t5_2
+ Filter: (blabel ~~ '%cd%'::text)
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%ab%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_2.a = t5_1.aid)
+ -> Hash Join
+ Hash Cond: (prt5_2.b = t5_2.bid)
+ -> Seq Scan on prt5_p1 prt5_2
+ -> Hash
+ -> Seq Scan on t5_2
+ Filter: (blabel ~~ '%cd%'::text)
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%ab%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_3.a = t5_1.aid)
+ -> Hash Join
+ Hash Cond: (prt5_3.b = t5_2.bid)
+ -> Seq Scan on prt5_p2 prt5_3
+ -> Hash
+ -> Seq Scan on t5_2
+ Filter: (blabel ~~ '%cd%'::text)
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%ab%'::text)
+(34 rows)
+
+-- unable to extract non-partitioned right relation
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 RIGHT JOIN t5_1 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+-----------------------------------------------
+ Hash Right Join
+ Hash Cond: (prt5.a = t5_1.aid)
+ Join Filter: (t5_1.alabel ~~ '%abc%'::text)
+ -> Append
+ -> Seq Scan on prt5_p0 prt5_1
+ -> Seq Scan on prt5_p1 prt5_2
+ -> Seq Scan on prt5_p2 prt5_3
+ -> Hash
+ -> Seq Scan on t5_1
+(9 rows)
+
+-- left side can be extracted, but no cost benefit
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 LEFT JOIN t5_1 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+-------------------------------------------------
+ Hash Left Join
+ Hash Cond: (prt5.a = t5_1.aid)
+ -> Append
+ -> Seq Scan on prt5_p0 prt5_1
+ -> Seq Scan on prt5_p1 prt5_2
+ -> Seq Scan on prt5_p2 prt5_3
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+(9 rows)
+
+-- validation of the results with/without asymmetric partition-wise join
+SELECT * INTO pg_temp.result01a
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+SELECT * INTO pg_temp.result02a
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+SET enable_partitionwise_join = off;
+SELECT * INTO pg_temp.result01b
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+SELECT * INTO pg_temp.result02b
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+SELECT * FROM pg_temp.result01a EXCEPT SELECT * FROM pg_temp.result01b;
+ hkey | a | b | aid | alabel
+------+---+---+-----+--------
+(0 rows)
+
+SELECT * FROM pg_temp.result02a EXCEPT SELECT * FROM pg_temp.result02b;
+ hkey | a | b | aid | alabel | bid | blabel
+------+---+---+-----+--------+-----+--------
+(0 rows)
+
+RESET max_parallel_workers_per_gather;
+SET enable_partitionwise_join = true;
+-- Parameterized path examples.
+CREATE TABLE prta (id integer, payload text) PARTITION BY HASH (id);
+CREATE TABLE prta1 PARTITION OF prta FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE prta2 PARTITION OF prta FOR VALUES WITH (modulus 2, remainder 1);
+CREATE UNIQUE INDEX ON prta1 (id);
+CREATE UNIQUE INDEX ON prta2 (id);
+INSERT INTO prta (id, payload)
+ (SELECT *, ('abc' || id)::text AS payload
+ FROM generate_series(1,1) AS id);
+CREATE TABLE prtb (id integer, payload text) PARTITION BY HASH (id);
+CREATE TABLE prtb1 PARTITION OF prtb FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE prtb2 PARTITION OF prtb FOR VALUES WITH (modulus 2, remainder 1);
+CREATE UNIQUE INDEX ON prtb1 (id);
+CREATE UNIQUE INDEX ON prtb2 (id);
+INSERT INTO prtb (id, payload)
+ (SELECT *, ('def' || id)::text AS payload
+ FROM generate_series(1,1000) AS id);
+CREATE TABLE e (id integer, payload text) PARTITION BY HASH (id);
+CREATE TABLE e1 PARTITION OF e FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE e2 PARTITION OF e FOR VALUES WITH (modulus 2, remainder 1);
+INSERT INTO e (id, payload)
+ (SELECT *, ('ghi' || id)::text AS payload
+ FROM generate_series(1,1000) AS id);
+CREATE UNIQUE INDEX ON e1 (id);
+CREATE UNIQUE INDEX ON e2 (id);
+ANALYZE prta,prtb,e;
+EXPLAIN (COSTS OFF)
+SELECT * FROM prta,prtb WHERE prta.id=prtb.id;
+ QUERY PLAN
+-----------------------------------------------------------
+ Append
+ -> Nested Loop
+ -> Seq Scan on prta1 prta_1
+ -> Index Scan using prtb1_id_idx on prtb1 prtb_1
+ Index Cond: (id = prta_1.id)
+ -> Nested Loop
+ -> Seq Scan on prta2 prta_2
+ -> Index Scan using prtb2_id_idx on prtb2 prtb_2
+ Index Cond: (id = prta_2.id)
+(9 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT * FROM prta,prtb,e WHERE prta.id=prtb.id AND prta.id=e.id;
+ QUERY PLAN
+-----------------------------------------------------------------
+ Append
+ -> Nested Loop
+ Join Filter: (prta_1.id = e_1.id)
+ -> Nested Loop
+ -> Seq Scan on prta1 prta_1
+ -> Index Scan using prtb1_id_idx on prtb1 prtb_1
+ Index Cond: (id = prta_1.id)
+ -> Index Scan using e1_id_idx on e1 e_1
+ Index Cond: (id = prtb_1.id)
+ -> Nested Loop
+ Join Filter: (prta_2.id = e_2.id)
+ -> Nested Loop
+ -> Seq Scan on prta2 prta_2
+ -> Index Scan using prtb2_id_idx on prtb2 prtb_2
+ Index Cond: (id = prta_2.id)
+ -> Index Scan using e2_id_idx on e2 e_2
+ Index Cond: (id = prtb_2.id)
+(17 rows)
+
-- semi join
EXPLAIN (COSTS OFF)
SELECT t1.* FROM prt1_adv t1 WHERE EXISTS (SELECT 1 FROM prt2_adv t2 WHERE t1.a = t2.b) AND t1.b = 0 ORDER BY t1.a;
diff --git a/src/test/regress/sql/partition_join.sql b/src/test/regress/sql/partition_join.sql
index d97b5b69ff..51b81ce3f8 100644
--- a/src/test/regress/sql/partition_join.sql
+++ b/src/test/regress/sql/partition_join.sql
@@ -536,6 +536,132 @@ EXPLAIN (COSTS OFF)
SELECT t1.a, t1.c, t2.b, t2.c FROM prt1_adv t1 INNER JOIN prt2_adv t2 ON (t1.a = t2.b) WHERE t1.b = 0 ORDER BY t1.a, t2.b;
SELECT t1.a, t1.c, t2.b, t2.c FROM prt1_adv t1 INNER JOIN prt2_adv t2 ON (t1.a = t2.b) WHERE t1.b = 0 ORDER BY t1.a, t2.b;
+--
+-- For asymmetric partition-wise join
+--
+CREATE TABLE prt5 (hkey int, a int, b int) PARTITION BY HASH(hkey);
+CREATE TABLE prt5_p0 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 0);
+CREATE TABLE prt5_p1 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 1);
+CREATE TABLE prt5_p2 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 2);
+CREATE TABLE prt6 (aid int, alabel text) PARTITION BY HASH(aid);
+CREATE TABLE prt6_p0 PARTITION OF prt6
+ FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE prt6_p1 PARTITION OF prt6
+ FOR VALUES WITH (modulus 2, remainder 1);
+CREATE TABLE t5_1 (aid int, alabel text);
+CREATE TABLE t5_2 (bid int, blabel text);
+
+INSERT INTO prt5 (SELECT x, (1000.0 * random())::int,
+ (1000.0 * random())::int
+ FROM generate_series(1,1000000) x);
+INSERT INTO t5_1 (SELECT x, md5(x::text) FROM generate_series(-200, 800) x);
+INSERT INTO t5_2 (SELECT x, md5(x::text) FROM generate_series(-200, 800) x);
+INSERT INTO prt6 (SELECT * FROM t5_1);
+
+VACUUM ANALYZE prt5,prt6,t5_1,t5_2;
+
+SET max_parallel_workers_per_gather = 0;
+
+-- Trivial asymmetric JOIN of partitioned table with a relation
+EXPLAIN (COSTS OFF)
+SELECT *
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+
+-- The same, but appended with UNION ALL
+EXPLAIN (COSTS OFF)
+SELECT * FROM (
+ (SELECT * FROM prt5_p0)
+ UNION ALL
+ (SELECT * FROM prt5_p1)
+ UNION ALL
+ (SELECT * FROM prt5_p2)
+ ) AS sq1
+JOIN t5_1 ON a = aid AND alabel like '%abc%';
+
+-- Don't allow asymmetric JOIN of two partitioned tables.
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 JOIN prt6 ON a = aid AND alabel like '%abc%';
+
+-- Check asymmetric JOIN with Subquery
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 JOIN (
+ SELECT * FROM prt6 LIMIT 1000
+) AS sq1 ON a = aid AND alabel like '%abc%';
+
+EXPLAIN (COSTS OFF)
+SELECT *
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+
+-- unable to extract non-partitioned right relation
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 RIGHT JOIN t5_1 ON a = aid AND alabel like '%abc%';
+-- left side can be extracted, but no cost benefit
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 LEFT JOIN t5_1 ON a = aid AND alabel like '%abc%';
+
+-- validation of the results with/without asymmetric partition-wise join
+SELECT * INTO pg_temp.result01a
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+
+SELECT * INTO pg_temp.result02a
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+
+SET enable_partitionwise_join = off;
+
+SELECT * INTO pg_temp.result01b
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+
+SELECT * INTO pg_temp.result02b
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+
+SELECT * FROM pg_temp.result01a EXCEPT SELECT * FROM pg_temp.result01b;
+SELECT * FROM pg_temp.result02a EXCEPT SELECT * FROM pg_temp.result02b;
+
+RESET max_parallel_workers_per_gather;
+SET enable_partitionwise_join = true;
+
+-- Parameterized path examples.
+CREATE TABLE prta (id integer, payload text) PARTITION BY HASH (id);
+CREATE TABLE prta1 PARTITION OF prta FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE prta2 PARTITION OF prta FOR VALUES WITH (modulus 2, remainder 1);
+CREATE UNIQUE INDEX ON prta1 (id);
+CREATE UNIQUE INDEX ON prta2 (id);
+INSERT INTO prta (id, payload)
+ (SELECT *, ('abc' || id)::text AS payload
+ FROM generate_series(1,1) AS id);
+
+CREATE TABLE prtb (id integer, payload text) PARTITION BY HASH (id);
+CREATE TABLE prtb1 PARTITION OF prtb FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE prtb2 PARTITION OF prtb FOR VALUES WITH (modulus 2, remainder 1);
+CREATE UNIQUE INDEX ON prtb1 (id);
+CREATE UNIQUE INDEX ON prtb2 (id);
+INSERT INTO prtb (id, payload)
+ (SELECT *, ('def' || id)::text AS payload
+ FROM generate_series(1,1000) AS id);
+
+CREATE TABLE e (id integer, payload text) PARTITION BY HASH (id);
+CREATE TABLE e1 PARTITION OF e FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE e2 PARTITION OF e FOR VALUES WITH (modulus 2, remainder 1);
+INSERT INTO e (id, payload)
+ (SELECT *, ('ghi' || id)::text AS payload
+ FROM generate_series(1,1000) AS id);
+CREATE UNIQUE INDEX ON e1 (id);
+CREATE UNIQUE INDEX ON e2 (id);
+
+ANALYZE prta,prtb,e;
+
+EXPLAIN (COSTS OFF)
+SELECT * FROM prta,prtb WHERE prta.id=prtb.id;
+
+EXPLAIN (COSTS OFF)
+SELECT * FROM prta,prtb,e WHERE prta.id=prtb.id AND prta.id=e.id;
+
-- semi join
EXPLAIN (COSTS OFF)
SELECT t1.* FROM prt1_adv t1 WHERE EXISTS (SELECT 1 FROM prt2_adv t2 WHERE t1.a = t2.b) AND t1.b = 0 ORDER BY t1.a;
--
2.31.1
^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: Asymmetric partition-wise JOIN
2019-08-12 06:03 Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-08-22 16:05 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-08-23 22:01 ` Re: Asymmetric partition-wise JOIN Thomas Munro <[email protected]>
2019-08-24 08:33 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-12-01 03:24 ` Re: Asymmetric partition-wise JOIN Michael Paquier <[email protected]>
2019-12-27 07:34 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2020-07-01 09:10 ` Re: Asymmetric partition-wise JOIN Daniel Gustafsson <[email protected]>
2020-08-21 06:02 ` Re: Asymmetric partition-wise JOIN Andrey V. Lepikhov <[email protected]>
2020-11-09 10:53 ` Re: Asymmetric partition-wise JOIN Anastasia Lubennikova <[email protected]>
2020-11-30 14:43 ` Re: Asymmetric partition-wise JOIN Anastasia Lubennikova <[email protected]>
2021-04-30 03:10 ` Re: Asymmetric partition-wise JOIN Andrey V. Lepikhov <[email protected]>
2021-05-27 04:27 ` Re: Asymmetric partition-wise JOIN Andrey Lepikhov <[email protected]>
2021-06-18 12:02 ` Re: Asymmetric partition-wise JOIN Alexander Pyhalov <[email protected]>
2021-07-05 09:57 ` Re: Asymmetric partition-wise JOIN Andrey Lepikhov <[email protected]>
2021-07-05 20:15 ` Re: Asymmetric partition-wise JOIN Zhihong Yu <[email protected]>
2021-07-06 09:28 ` Re: Asymmetric partition-wise JOIN Andrey Lepikhov <[email protected]>
@ 2021-07-06 13:09 ` Alexander Pyhalov <[email protected]>
0 siblings, 0 replies; 28+ messages in thread
From: Alexander Pyhalov @ 2021-07-06 13:09 UTC (permalink / raw)
To: Andrey Lepikhov <[email protected]>; +Cc: Zhihong Yu <[email protected]>; Anastasia Lubennikova <[email protected]>; Daniel Gustafsson <[email protected]>; Kohei KaiGai <[email protected]>; Michael Paquier <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers
Andrey Lepikhov писал 2021-07-06 12:28:
> On 5/7/21 23:15, Zhihong Yu wrote:
>> On Mon, Jul 5, 2021 at 2:57 AM Andrey Lepikhov
>> <[email protected] <mailto:[email protected]>> wrote:
>> + * Can't imagine situation when join relation already
>> exists. But in
>> + * the 'partition_join' regression test it happens.
>> + * It may be an indicator of possible problems.
>> Should a log be added in the above case ?
> I made additional analysis of this branch of code. This situation can
> happen in the case of one child or if we join two plane tables with
> partitioned. Both situations are legal and I think we don't needed to
> add any log message here.
> Other mistakes were fixed.
Hi.
Small typo in comment in src/backend/optimizer/plan/setrefs.c:
281
282 /*
283 * Adjust RT indexes of AppendRelInfos and add to final
appendrels list.
284 * The AppendRelInfos are copied, because as a part of a
subplan its could
285 * be visited many times in the case of asymmetric join.
286 */
287 foreach(lc, root->append_rel_list)
288 {
its -> it (or they) ?
--
Best regards,
Alexander Pyhalov,
Postgres Professional
^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: Asymmetric partition-wise JOIN
2019-08-12 06:03 Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-08-22 16:05 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-08-23 22:01 ` Re: Asymmetric partition-wise JOIN Thomas Munro <[email protected]>
2019-08-24 08:33 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-12-01 03:24 ` Re: Asymmetric partition-wise JOIN Michael Paquier <[email protected]>
2019-12-27 07:34 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2020-07-01 09:10 ` Re: Asymmetric partition-wise JOIN Daniel Gustafsson <[email protected]>
2020-08-21 06:02 ` Re: Asymmetric partition-wise JOIN Andrey V. Lepikhov <[email protected]>
2020-11-09 10:53 ` Re: Asymmetric partition-wise JOIN Anastasia Lubennikova <[email protected]>
2020-11-30 14:43 ` Re: Asymmetric partition-wise JOIN Anastasia Lubennikova <[email protected]>
2021-04-30 03:10 ` Re: Asymmetric partition-wise JOIN Andrey V. Lepikhov <[email protected]>
2021-05-27 04:27 ` Re: Asymmetric partition-wise JOIN Andrey Lepikhov <[email protected]>
2021-06-18 12:02 ` Re: Asymmetric partition-wise JOIN Alexander Pyhalov <[email protected]>
2021-07-05 09:57 ` Re: Asymmetric partition-wise JOIN Andrey Lepikhov <[email protected]>
2021-07-05 20:15 ` Re: Asymmetric partition-wise JOIN Zhihong Yu <[email protected]>
@ 2021-07-15 06:32 ` Andrey Lepikhov <[email protected]>
2021-07-15 13:16 ` Re: Asymmetric partition-wise JOIN Ibrar Ahmed <[email protected]>
1 sibling, 1 reply; 28+ messages in thread
From: Andrey Lepikhov @ 2021-07-15 06:32 UTC (permalink / raw)
To: Zhihong Yu <[email protected]>; +Cc: Alexander Pyhalov <[email protected]>; Anastasia Lubennikova <[email protected]>; Daniel Gustafsson <[email protected]>; Kohei KaiGai <[email protected]>; Michael Paquier <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers
On 5/7/21 23:15, Zhihong Yu wrote:
> On Mon, Jul 5, 2021 at 2:57 AM Andrey Lepikhov
> <[email protected] <mailto:[email protected]>> wrote:
> + * Can't imagine situation when join relation already
> exists. But in
> + * the 'partition_join' regression test it happens.
> + * It may be an indicator of possible problems.
>
> Should a log be added in the above case ?
I worked more on this case and found more serious mistake. During
population of additional paths on the existed RelOptInfo we can remove
some previously generated paths that pointed from a higher-level list of
subplans and it could cause to lost of subplan links. I prohibit such
situation (you can read comments in the new version of the patch).
Also, choosing of a cheapest path after appendrel creation was added.
Unstable tests were fixed.
--
regards,
Andrey Lepikhov
Postgres Professional
From 7b05d8535b11ed7b5c43ed591d9aeea955f2c2ed Mon Sep 17 00:00:00 2001
From: Andrey Lepikhov <[email protected]>
Date: Fri, 2 Apr 2021 11:02:20 +0500
Subject: [PATCH] Asymmetric partitionwise join.
Teach optimizer to consider partitionwise join of non-partitioned
table with each partition of partitioned table.
Disallow asymmetric machinery for joining of two partitioned (or appended)
relations because it could cause huge consumption of CPU and memory
during reparameterization of NestLoop path.
---
src/backend/optimizer/path/joinpath.c | 9 +
src/backend/optimizer/path/joinrels.c | 187 +++++++++++
src/backend/optimizer/plan/setrefs.c | 17 +-
src/backend/optimizer/util/appendinfo.c | 37 ++-
src/backend/optimizer/util/pathnode.c | 9 +-
src/backend/optimizer/util/relnode.c | 19 +-
src/include/optimizer/paths.h | 7 +-
src/test/regress/expected/partition_join.out | 332 +++++++++++++++++++
src/test/regress/sql/partition_join.sql | 133 ++++++++
9 files changed, 720 insertions(+), 30 deletions(-)
diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c
index 6407ede12a..32618ebbd5 100644
--- a/src/backend/optimizer/path/joinpath.c
+++ b/src/backend/optimizer/path/joinpath.c
@@ -335,6 +335,15 @@ add_paths_to_joinrel(PlannerInfo *root,
if (set_join_pathlist_hook)
set_join_pathlist_hook(root, joinrel, outerrel, innerrel,
jointype, &extra);
+
+ /*
+ * 7. If outer relation is delivered from partition-tables, consider
+ * distributing inner relation to every partition-leaf prior to
+ * append these leafs.
+ */
+ try_asymmetric_partitionwise_join(root, joinrel,
+ outerrel, innerrel,
+ jointype, &extra);
}
/*
diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c
index 0dbe2ac726..c8db6f0969 100644
--- a/src/backend/optimizer/path/joinrels.c
+++ b/src/backend/optimizer/path/joinrels.c
@@ -16,6 +16,7 @@
#include "miscadmin.h"
#include "optimizer/appendinfo.h"
+#include "optimizer/cost.h"
#include "optimizer/joininfo.h"
#include "optimizer/pathnode.h"
#include "optimizer/paths.h"
@@ -1551,6 +1552,192 @@ try_partitionwise_join(PlannerInfo *root, RelOptInfo *rel1, RelOptInfo *rel2,
}
}
+/*
+ * Build RelOptInfo on JOIN of each partition of the outer relation and the inner
+ * relation. Return List of such RelOptInfo's. Return NIL, if at least one of
+ * these JOINs is impossible to build.
+ */
+static List *
+extract_asymmetric_partitionwise_subjoin(PlannerInfo *root,
+ RelOptInfo *joinrel,
+ AppendPath *append_path,
+ RelOptInfo *inner_rel,
+ JoinType jointype,
+ JoinPathExtraData *extra)
+{
+ List *result = NIL;
+ ListCell *lc;
+
+ foreach (lc, append_path->subpaths)
+ {
+ Path *child_path = lfirst(lc);
+ RelOptInfo *child_rel = child_path->parent;
+ Relids child_joinrelids;
+ Relids parent_relids;
+ RelOptInfo *child_joinrel;
+ SpecialJoinInfo *child_sjinfo;
+ List *child_restrictlist;
+
+ child_joinrelids = bms_union(child_rel->relids, inner_rel->relids);
+ parent_relids = bms_union(append_path->path.parent->relids,
+ inner_rel->relids);
+
+ child_sjinfo = build_child_join_sjinfo(root, extra->sjinfo,
+ child_rel->relids,
+ inner_rel->relids);
+ child_restrictlist = (List *)
+ adjust_appendrel_attrs_multilevel(root, (Node *)extra->restrictlist,
+ child_joinrelids, parent_relids);
+
+ child_joinrel = find_join_rel(root, child_joinrelids);
+ if (!child_joinrel)
+ child_joinrel = build_child_join_rel(root,
+ child_rel,
+ inner_rel,
+ joinrel,
+ child_restrictlist,
+ child_sjinfo,
+ jointype);
+ else
+ {
+ /*
+ * The join relation already exists. For example, it could happen if
+ * we join two plane tables with partitioned table(s).
+ * Populating this join with additional paths could push out some
+ * previously added paths which could be pointed in a subplans list
+ * of an higher level append.
+ * Of course, we could save such paths before generating new. But it
+ * can increase too much the number of paths in complex queries. It
+ * can be a task for future work.
+ */
+ return NIL;
+ }
+
+ populate_joinrel_with_paths(root,
+ child_rel,
+ inner_rel,
+ child_joinrel,
+ child_sjinfo,
+ child_restrictlist);
+
+ /* Give up if asymmetric partition-wise join is not available */
+ if (child_joinrel->pathlist == NIL)
+ return NIL;
+
+ set_cheapest(child_joinrel);
+ result = lappend(result, child_joinrel);
+ }
+ return result;
+}
+
+static bool
+is_asymmetric_join_feasible(PlannerInfo *root,
+ RelOptInfo *outer_rel,
+ RelOptInfo *inner_rel,
+ JoinType jointype)
+{
+ ListCell *lc;
+
+ if (jointype != JOIN_INNER && jointype != JOIN_LEFT)
+ return false;
+
+ if (IS_OTHER_REL(outer_rel) || IS_OTHER_REL(inner_rel))
+ return false;
+
+ /* Disallow recursive usage of asymmertic join machinery */
+ if (root->join_rel_level == NULL)
+ return false;
+
+ /*
+ * Don't allow asymmetric JOIN of two append subplans.
+ * In the case of a parameterized NL join, a reparameterization procedure
+ * will lead to large memory allocations and a CPU consumption:
+ * each reparameterization will induce subpath duplication, creating new
+ * ParamPathInfo instance and increasing of ppilist up to number of
+ * partitions in the inner. Also, if we have many partitions, each bitmapset
+ * variable will be large and many leaks of such variable (caused by relid
+ * replacement) will highly increase memory consumption.
+ * So, we deny such paths for now.
+ */
+ foreach(lc, inner_rel->pathlist)
+ {
+ if (IsA(lfirst(lc), AppendPath))
+ return false;
+ }
+
+ return true;
+}
+
+void
+try_asymmetric_partitionwise_join(PlannerInfo *root,
+ RelOptInfo *joinrel,
+ RelOptInfo *outer_rel,
+ RelOptInfo *inner_rel,
+ JoinType jointype,
+ JoinPathExtraData *extra)
+{
+ ListCell *lc;
+
+ /*
+ * Try this kind of paths if we allow complex partitionwise joins and we know
+ * we can build this join safely.
+ */
+ if (!enable_partitionwise_join ||
+ !is_asymmetric_join_feasible(root, outer_rel, inner_rel, jointype))
+ return;
+
+ foreach (lc, outer_rel->pathlist)
+ {
+ AppendPath *append_path = lfirst(lc);
+
+ /*
+ * We assume this pathlist keeps at least one AppendPath that
+ * represents partitioned table-scan, symmetric or asymmetric
+ * partition-wise join. Asymmetric join isn't needed if the append node
+ * has only one child.
+ */
+ if (IsA(append_path, AppendPath) &&
+ list_length(append_path->subpaths) > 1)
+ {
+ List **join_rel_level_saved;
+ List *live_childrels = NIL;
+
+ join_rel_level_saved = root->join_rel_level;
+ PG_TRY();
+ {
+ /* temporary disables "dynamic programming" algorithm */
+ root->join_rel_level = NULL;
+
+ live_childrels =
+ extract_asymmetric_partitionwise_subjoin(root,
+ joinrel,
+ append_path,
+ inner_rel,
+ jointype,
+ extra);
+ }
+ PG_FINALLY();
+ {
+ root->join_rel_level = join_rel_level_saved;
+ }
+ PG_END_TRY();
+
+ if (live_childrels != NIL)
+ {
+ /*
+ * Add new append relation. We must choose cheapest paths after
+ * this operation because the pathlist possibly contains
+ * joinrels and appendrels that can be suboptimal.
+ */
+ add_paths_to_append_rel(root, joinrel, live_childrels);
+ set_cheapest(joinrel);
+ }
+
+ break;
+ }
+ }
+}
+
/*
* Construct the SpecialJoinInfo for a child-join by translating
* SpecialJoinInfo for the join between parents. left_relids and right_relids
diff --git a/src/backend/optimizer/plan/setrefs.c b/src/backend/optimizer/plan/setrefs.c
index 26f6872b4b..6ffa68efba 100644
--- a/src/backend/optimizer/plan/setrefs.c
+++ b/src/backend/optimizer/plan/setrefs.c
@@ -281,24 +281,29 @@ set_plan_references(PlannerInfo *root, Plan *plan)
/*
* Adjust RT indexes of AppendRelInfos and add to final appendrels list.
- * We assume the AppendRelInfos were built during planning and don't need
- * to be copied.
+ * The AppendRelInfos are copied, because as a part of a subplan they could
+ * be visited many times in the case of asymmetric join.
*/
foreach(lc, root->append_rel_list)
{
AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc);
+ AppendRelInfo *newappinfo;
+
+ /* flat copy is enough since all valuable fields are scalars */
+ newappinfo = (AppendRelInfo *) palloc(sizeof(AppendRelInfo));
+ memcpy(newappinfo, appinfo, sizeof(AppendRelInfo));
/* adjust RT indexes */
- appinfo->parent_relid += rtoffset;
- appinfo->child_relid += rtoffset;
+ newappinfo->parent_relid += rtoffset;
+ newappinfo->child_relid += rtoffset;
/*
* Rather than adjust the translated_vars entries, just drop 'em.
* Neither the executor nor EXPLAIN currently need that data.
*/
- appinfo->translated_vars = NIL;
+ newappinfo->translated_vars = NIL;
- glob->appendRelations = lappend(glob->appendRelations, appinfo);
+ glob->appendRelations = lappend(glob->appendRelations, newappinfo);
}
/* Now fix the Plan tree */
diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c
index af46f581ac..ec8dee99b0 100644
--- a/src/backend/optimizer/util/appendinfo.c
+++ b/src/backend/optimizer/util/appendinfo.c
@@ -202,7 +202,9 @@ adjust_appendrel_attrs(PlannerInfo *root, Node *node, int nappinfos,
context.appinfos = appinfos;
/* If there's nothing to adjust, don't call this function. */
- Assert(nappinfos >= 1 && appinfos != NULL);
+ /* If there's nothing to adjust, just return a duplication */
+ if (nappinfos == 0)
+ return copyObject(node);
/* Should never be translating a Query tree. */
Assert(node == NULL || !IsA(node, Query));
@@ -489,12 +491,10 @@ adjust_appendrel_attrs_multilevel(PlannerInfo *root, Node *node,
Relids child_relids,
Relids top_parent_relids)
{
- AppendRelInfo **appinfos;
- Bitmapset *parent_relids = NULL;
- int nappinfos;
- int cnt;
-
- Assert(bms_num_members(child_relids) == bms_num_members(top_parent_relids));
+ AppendRelInfo **appinfos;
+ Relids parent_relids = NULL;
+ int nappinfos;
+ int cnt;
appinfos = find_appinfos_by_relids(root, child_relids, &nappinfos);
@@ -506,8 +506,13 @@ adjust_appendrel_attrs_multilevel(PlannerInfo *root, Node *node,
parent_relids = bms_add_member(parent_relids, appinfo->parent_relid);
}
- /* Recurse if immediate parent is not the top parent. */
- if (!bms_equal(parent_relids, top_parent_relids))
+ /*
+ * Recurse if immediate parent is not the top parent. Keep in mind that in a
+ * case of asymmetric JOIN top_parent_relids can contain relids which aren't
+ * part of an append node.
+ */
+ if (!bms_equal(parent_relids, top_parent_relids) &&
+ !bms_is_subset(parent_relids, top_parent_relids))
node = adjust_appendrel_attrs_multilevel(root, node, parent_relids,
top_parent_relids);
@@ -515,6 +520,7 @@ adjust_appendrel_attrs_multilevel(PlannerInfo *root, Node *node,
node = adjust_appendrel_attrs(root, node, nappinfos, appinfos);
pfree(appinfos);
+ pfree(parent_relids);
return node;
}
@@ -565,6 +571,7 @@ adjust_child_relids_multilevel(PlannerInfo *root, Relids relids,
AppendRelInfo **appinfos;
int nappinfos;
Relids parent_relids = NULL;
+ Relids normal_relids = NULL;
Relids result;
Relids tmp_result = NULL;
int cnt;
@@ -579,12 +586,17 @@ adjust_child_relids_multilevel(PlannerInfo *root, Relids relids,
appinfos = find_appinfos_by_relids(root, child_relids, &nappinfos);
/* Construct relids set for the immediate parent of the given child. */
+ normal_relids = bms_copy(child_relids);
for (cnt = 0; cnt < nappinfos; cnt++)
{
AppendRelInfo *appinfo = appinfos[cnt];
parent_relids = bms_add_member(parent_relids, appinfo->parent_relid);
+ normal_relids = bms_del_member(normal_relids, appinfo->child_relid);
}
+ parent_relids = bms_union(parent_relids, normal_relids);
+ if (normal_relids)
+ bms_free(normal_relids);
/* Recurse if immediate parent is not the top parent. */
if (!bms_equal(parent_relids, top_parent_relids))
@@ -715,11 +727,11 @@ AppendRelInfo **
find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos)
{
AppendRelInfo **appinfos;
+ int nrooms = bms_num_members(relids);
int cnt = 0;
int i;
- *nappinfos = bms_num_members(relids);
- appinfos = (AppendRelInfo **) palloc(sizeof(AppendRelInfo *) * *nappinfos);
+ appinfos = (AppendRelInfo **) palloc(sizeof(AppendRelInfo *) * nrooms);
i = -1;
while ((i = bms_next_member(relids, i)) >= 0)
@@ -727,10 +739,11 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos)
AppendRelInfo *appinfo = root->append_rel_array[i];
if (!appinfo)
- elog(ERROR, "child rel %d not found in append_rel_array", i);
+ continue;
appinfos[cnt++] = appinfo;
}
+ *nappinfos = cnt;
return appinfos;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index 0c94cbe767..e4828b3647 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -4224,7 +4224,14 @@ do { \
MemoryContextSwitchTo(oldcontext);
}
- bms_free(required_outer);
+
+ /*
+ * If adjust_child_relids_multilevel don't do replacements it returns
+ * the original set, not a copy. It is possible in the case of asymmetric
+ * JOIN and child_rel->relids contains relids only of plane relations.
+ */
+ if (required_outer != old_ppi->ppi_req_outer)
+ bms_free(required_outer);
new_path->param_info = new_ppi;
diff --git a/src/backend/optimizer/util/relnode.c b/src/backend/optimizer/util/relnode.c
index e105a4d5f1..bb15fb2716 100644
--- a/src/backend/optimizer/util/relnode.c
+++ b/src/backend/optimizer/util/relnode.c
@@ -790,11 +790,8 @@ build_child_join_rel(PlannerInfo *root, RelOptInfo *outer_rel,
AppendRelInfo **appinfos;
int nappinfos;
- /* Only joins between "other" relations land here. */
- Assert(IS_OTHER_REL(outer_rel) && IS_OTHER_REL(inner_rel));
-
- /* The parent joinrel should have consider_partitionwise_join set. */
- Assert(parent_joinrel->consider_partitionwise_join);
+ /* Either of relations must be "other" relation at least. */
+ Assert(IS_OTHER_REL(outer_rel) || IS_OTHER_REL(inner_rel));
joinrel->reloptkind = RELOPT_OTHER_JOINREL;
joinrel->relids = bms_union(outer_rel->relids, inner_rel->relids);
@@ -851,8 +848,11 @@ build_child_join_rel(PlannerInfo *root, RelOptInfo *outer_rel,
joinrel->partexprs = NULL;
joinrel->nullable_partexprs = NULL;
- joinrel->top_parent_relids = bms_union(outer_rel->top_parent_relids,
- inner_rel->top_parent_relids);
+ joinrel->top_parent_relids =
+ bms_union(IS_OTHER_REL(outer_rel) ?
+ outer_rel->top_parent_relids : outer_rel->relids,
+ IS_OTHER_REL(inner_rel) ?
+ inner_rel->top_parent_relids : inner_rel->relids);
/* Compute information relevant to foreign relations. */
set_foreign_rel_properties(joinrel, outer_rel, inner_rel);
@@ -2033,9 +2033,8 @@ build_child_join_reltarget(PlannerInfo *root,
{
/* Build the targetlist */
childrel->reltarget->exprs = (List *)
- adjust_appendrel_attrs(root,
- (Node *) parentrel->reltarget->exprs,
- nappinfos, appinfos);
+ adjust_appendrel_attrs_multilevel(root, (Node *)parentrel->reltarget->exprs,
+ childrel->relids, parentrel->relids);
/* Set the cost and width fields */
childrel->reltarget->cost.startup = parentrel->reltarget->cost.startup;
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index f1d111063c..a0106dc539 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -109,7 +109,12 @@ extern bool have_join_order_restriction(PlannerInfo *root,
extern bool have_dangerous_phv(PlannerInfo *root,
Relids outer_relids, Relids inner_params);
extern void mark_dummy_rel(RelOptInfo *rel);
-
+extern void try_asymmetric_partitionwise_join(PlannerInfo *root,
+ RelOptInfo *joinrel,
+ RelOptInfo *outer_rel,
+ RelOptInfo *inner_rel,
+ JoinType jointype,
+ JoinPathExtraData *extra);
/*
* equivclass.c
* routines for managing EquivalenceClasses
diff --git a/src/test/regress/expected/partition_join.out b/src/test/regress/expected/partition_join.out
index 27f7525b3e..753b3de47f 100644
--- a/src/test/regress/expected/partition_join.out
+++ b/src/test/regress/expected/partition_join.out
@@ -2320,6 +2320,338 @@ SELECT t1.a, t1.c, t2.b, t2.c FROM prt1_adv t1 INNER JOIN prt2_adv t2 ON (t1.a =
375 | 0375 | 375 | 0375
(8 rows)
+--
+-- For asymmetric partition-wise join
+--
+CREATE TABLE prt5 (hkey int, a int, b int) PARTITION BY HASH(hkey);
+CREATE TABLE prt5_p0 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 0);
+CREATE TABLE prt5_p1 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 1);
+CREATE TABLE prt5_p2 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 2);
+CREATE TABLE prt6 (aid int, alabel text) PARTITION BY HASH(aid);
+CREATE TABLE prt6_p0 PARTITION OF prt6
+ FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE prt6_p1 PARTITION OF prt6
+ FOR VALUES WITH (modulus 2, remainder 1);
+CREATE TABLE t5_1 (aid int, alabel text);
+CREATE TABLE t5_2 (bid int, blabel text);
+INSERT INTO prt5 (SELECT x, (x % 1000)::int,
+ ((x+1) % 1000)::int
+ FROM generate_series(1,1000000) x);
+INSERT INTO t5_1 (SELECT x, md5(x::text) FROM generate_series(-200, 800) x);
+INSERT INTO t5_2 (SELECT x, md5(x::text) FROM generate_series(-200, 800) x);
+INSERT INTO prt6 (SELECT * FROM t5_1);
+VACUUM ANALYZE prt5,prt6,t5_1,t5_2;
+SET max_parallel_workers_per_gather = 0;
+-- Trivial asymmetric JOIN of partitioned table with a relation
+EXPLAIN (COSTS OFF)
+SELECT *
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+-------------------------------------------------------
+ Append
+ -> Hash Join
+ Hash Cond: (prt5_1.a = t5_1.aid)
+ -> Seq Scan on prt5_p0 prt5_1
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_2.a = t5_1.aid)
+ -> Seq Scan on prt5_p1 prt5_2
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_3.a = t5_1.aid)
+ -> Seq Scan on prt5_p2 prt5_3
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+(19 rows)
+
+-- The same, but appended with UNION ALL
+EXPLAIN (COSTS OFF)
+SELECT * FROM (
+ (SELECT * FROM prt5_p0)
+ UNION ALL
+ (SELECT * FROM prt5_p1)
+ UNION ALL
+ (SELECT * FROM prt5_p2)
+ ) AS sq1
+JOIN t5_1 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+-------------------------------------------------------
+ Append
+ -> Hash Join
+ Hash Cond: (prt5_p0.a = t5_1.aid)
+ -> Seq Scan on prt5_p0
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_p1.a = t5_1.aid)
+ -> Seq Scan on prt5_p1
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_p2.a = t5_1.aid)
+ -> Seq Scan on prt5_p2
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+(19 rows)
+
+-- Don't allow asymmetric JOIN of two partitioned tables.
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM prt5 JOIN prt6 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+-------------------------------------------------------------
+ Aggregate
+ -> Hash Join
+ Hash Cond: (prt5.a = prt6.aid)
+ -> Append
+ -> Seq Scan on prt5_p0 prt5_1
+ -> Seq Scan on prt5_p1 prt5_2
+ -> Seq Scan on prt5_p2 prt5_3
+ -> Hash
+ -> Append
+ -> Seq Scan on prt6_p0 prt6_1
+ Filter: (alabel ~~ '%abc%'::text)
+ -> Seq Scan on prt6_p1 prt6_2
+ Filter: (alabel ~~ '%abc%'::text)
+(13 rows)
+
+SELECT count(*) FROM prt5 JOIN prt6 ON a = aid AND alabel like '%abc%';
+ count
+-------
+ 4000
+(1 row)
+
+-- Check asymmetric JOIN with Subquery
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM prt5 JOIN (
+ SELECT * FROM prt6 LIMIT 1000
+) AS sq1 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+----------------------------------------------------------------------
+ Aggregate
+ -> Append
+ -> Hash Join
+ Hash Cond: (prt5_1.a = sq1.aid)
+ -> Seq Scan on prt5_p0 prt5_1
+ -> Hash
+ -> Subquery Scan on sq1
+ Filter: (sq1.alabel ~~ '%abc%'::text)
+ -> Limit
+ -> Append
+ -> Seq Scan on prt6_p0 prt6_1
+ -> Seq Scan on prt6_p1 prt6_2
+ -> Hash Join
+ Hash Cond: (prt5_2.a = sq1.aid)
+ -> Seq Scan on prt5_p1 prt5_2
+ -> Hash
+ -> Subquery Scan on sq1
+ Filter: (sq1.alabel ~~ '%abc%'::text)
+ -> Limit
+ -> Append
+ -> Seq Scan on prt6_p0 prt6_4
+ -> Seq Scan on prt6_p1 prt6_5
+ -> Hash Join
+ Hash Cond: (prt5_3.a = sq1.aid)
+ -> Seq Scan on prt5_p2 prt5_3
+ -> Hash
+ -> Subquery Scan on sq1
+ Filter: (sq1.alabel ~~ '%abc%'::text)
+ -> Limit
+ -> Append
+ -> Seq Scan on prt6_p0 prt6_7
+ -> Seq Scan on prt6_p1 prt6_8
+(32 rows)
+
+SELECT count(*) FROM prt5 JOIN (SELECT * FROM prt6 LIMIT 1000) AS sq1
+ ON a = aid AND alabel like '%abc%';
+ count
+-------
+ 4000
+(1 row)
+
+-- Asymmetric JOIN of two plane tables and one partitioned
+EXPLAIN (COSTS OFF)
+SELECT count(*)
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+ QUERY PLAN
+------------------------------------------------------------------
+ Aggregate
+ -> Append
+ -> Hash Join
+ Hash Cond: (prt5_1.a = t5_1.aid)
+ -> Hash Join
+ Hash Cond: (prt5_1.b = t5_2.bid)
+ -> Seq Scan on prt5_p0 prt5_1
+ -> Hash
+ -> Seq Scan on t5_2
+ Filter: (blabel ~~ '%cd%'::text)
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%ab%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_2.a = t5_1.aid)
+ -> Hash Join
+ Hash Cond: (prt5_2.b = t5_2.bid)
+ -> Seq Scan on prt5_p1 prt5_2
+ -> Hash
+ -> Seq Scan on t5_2
+ Filter: (blabel ~~ '%cd%'::text)
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%ab%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_3.a = t5_1.aid)
+ -> Hash Join
+ Hash Cond: (prt5_3.b = t5_2.bid)
+ -> Seq Scan on prt5_p2 prt5_3
+ -> Hash
+ -> Seq Scan on t5_2
+ Filter: (blabel ~~ '%cd%'::text)
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%ab%'::text)
+(35 rows)
+
+SELECT count(*)
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+ count
+-------
+ 10000
+(1 row)
+
+-- unable to extract non-partitioned right relation
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 RIGHT JOIN t5_1 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+-----------------------------------------------
+ Hash Right Join
+ Hash Cond: (prt5.a = t5_1.aid)
+ Join Filter: (t5_1.alabel ~~ '%abc%'::text)
+ -> Append
+ -> Seq Scan on prt5_p0 prt5_1
+ -> Seq Scan on prt5_p1 prt5_2
+ -> Seq Scan on prt5_p2 prt5_3
+ -> Hash
+ -> Seq Scan on t5_1
+(9 rows)
+
+-- left side can be extracted, but no cost benefit
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 LEFT JOIN t5_1 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+-------------------------------------------------
+ Hash Left Join
+ Hash Cond: (prt5.a = t5_1.aid)
+ -> Append
+ -> Seq Scan on prt5_p0 prt5_1
+ -> Seq Scan on prt5_p1 prt5_2
+ -> Seq Scan on prt5_p2 prt5_3
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+(9 rows)
+
+SET enable_partitionwise_join = on;
+-- validation of the results with/without asymmetric partition-wise join
+SELECT * INTO pg_temp.result01a
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+SELECT * INTO pg_temp.result02a
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+SET enable_partitionwise_join = off;
+SELECT * INTO pg_temp.result01b
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+SELECT * INTO pg_temp.result02b
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+SELECT * FROM pg_temp.result01a EXCEPT SELECT * FROM pg_temp.result01b;
+ hkey | a | b | aid | alabel
+------+---+---+-----+--------
+(0 rows)
+
+SELECT * FROM pg_temp.result02a EXCEPT SELECT * FROM pg_temp.result02b;
+ hkey | a | b | aid | alabel | bid | blabel
+------+---+---+-----+--------+-----+--------
+(0 rows)
+
+RESET max_parallel_workers_per_gather;
+SET enable_partitionwise_join = true;
+-- Parameterized path examples.
+CREATE TABLE prta (id integer, payload text) PARTITION BY HASH (id);
+CREATE TABLE prta1 PARTITION OF prta FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE prta2 PARTITION OF prta FOR VALUES WITH (modulus 2, remainder 1);
+CREATE UNIQUE INDEX ON prta1 (id);
+CREATE UNIQUE INDEX ON prta2 (id);
+INSERT INTO prta (id, payload)
+ (SELECT *, ('abc' || id)::text AS payload
+ FROM generate_series(1,1) AS id);
+CREATE TABLE prtb (id integer, payload text) PARTITION BY HASH (id);
+CREATE TABLE prtb1 PARTITION OF prtb FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE prtb2 PARTITION OF prtb FOR VALUES WITH (modulus 2, remainder 1);
+CREATE UNIQUE INDEX ON prtb1 (id);
+CREATE UNIQUE INDEX ON prtb2 (id);
+INSERT INTO prtb (id, payload)
+ (SELECT *, ('def' || id)::text AS payload
+ FROM generate_series(1,1000) AS id);
+CREATE TABLE e (id integer, payload text) PARTITION BY HASH (id);
+CREATE TABLE e1 PARTITION OF e FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE e2 PARTITION OF e FOR VALUES WITH (modulus 2, remainder 1);
+INSERT INTO e (id, payload)
+ (SELECT *, ('ghi' || id)::text AS payload
+ FROM generate_series(1,1000) AS id);
+CREATE UNIQUE INDEX ON e1 (id);
+CREATE UNIQUE INDEX ON e2 (id);
+ANALYZE prta,prtb,e;
+EXPLAIN (COSTS OFF)
+SELECT * FROM prta,prtb WHERE prta.id=prtb.id;
+ QUERY PLAN
+-----------------------------------------------------------
+ Append
+ -> Nested Loop
+ -> Seq Scan on prta1 prta_1
+ -> Index Scan using prtb1_id_idx on prtb1 prtb_1
+ Index Cond: (id = prta_1.id)
+ -> Nested Loop
+ -> Seq Scan on prta2 prta_2
+ -> Index Scan using prtb2_id_idx on prtb2 prtb_2
+ Index Cond: (id = prta_2.id)
+(9 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT * FROM prta,prtb,e WHERE prta.id=prtb.id AND prta.id=e.id;
+ QUERY PLAN
+-----------------------------------------------------------------
+ Append
+ -> Nested Loop
+ Join Filter: (prta_1.id = e_1.id)
+ -> Nested Loop
+ -> Seq Scan on prta1 prta_1
+ -> Index Scan using prtb1_id_idx on prtb1 prtb_1
+ Index Cond: (id = prta_1.id)
+ -> Index Scan using e1_id_idx on e1 e_1
+ Index Cond: (id = prtb_1.id)
+ -> Nested Loop
+ Join Filter: (prta_2.id = e_2.id)
+ -> Nested Loop
+ -> Seq Scan on prta2 prta_2
+ -> Index Scan using prtb2_id_idx on prtb2 prtb_2
+ Index Cond: (id = prta_2.id)
+ -> Index Scan using e2_id_idx on e2 e_2
+ Index Cond: (id = prtb_2.id)
+(17 rows)
+
-- semi join
EXPLAIN (COSTS OFF)
SELECT t1.* FROM prt1_adv t1 WHERE EXISTS (SELECT 1 FROM prt2_adv t2 WHERE t1.a = t2.b) AND t1.b = 0 ORDER BY t1.a;
diff --git a/src/test/regress/sql/partition_join.sql b/src/test/regress/sql/partition_join.sql
index d97b5b69ff..aca7f885a3 100644
--- a/src/test/regress/sql/partition_join.sql
+++ b/src/test/regress/sql/partition_join.sql
@@ -536,6 +536,139 @@ EXPLAIN (COSTS OFF)
SELECT t1.a, t1.c, t2.b, t2.c FROM prt1_adv t1 INNER JOIN prt2_adv t2 ON (t1.a = t2.b) WHERE t1.b = 0 ORDER BY t1.a, t2.b;
SELECT t1.a, t1.c, t2.b, t2.c FROM prt1_adv t1 INNER JOIN prt2_adv t2 ON (t1.a = t2.b) WHERE t1.b = 0 ORDER BY t1.a, t2.b;
+--
+-- For asymmetric partition-wise join
+--
+CREATE TABLE prt5 (hkey int, a int, b int) PARTITION BY HASH(hkey);
+CREATE TABLE prt5_p0 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 0);
+CREATE TABLE prt5_p1 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 1);
+CREATE TABLE prt5_p2 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 2);
+CREATE TABLE prt6 (aid int, alabel text) PARTITION BY HASH(aid);
+CREATE TABLE prt6_p0 PARTITION OF prt6
+ FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE prt6_p1 PARTITION OF prt6
+ FOR VALUES WITH (modulus 2, remainder 1);
+CREATE TABLE t5_1 (aid int, alabel text);
+CREATE TABLE t5_2 (bid int, blabel text);
+
+INSERT INTO prt5 (SELECT x, (x % 1000)::int,
+ ((x+1) % 1000)::int
+ FROM generate_series(1,1000000) x);
+INSERT INTO t5_1 (SELECT x, md5(x::text) FROM generate_series(-200, 800) x);
+INSERT INTO t5_2 (SELECT x, md5(x::text) FROM generate_series(-200, 800) x);
+INSERT INTO prt6 (SELECT * FROM t5_1);
+
+VACUUM ANALYZE prt5,prt6,t5_1,t5_2;
+
+SET max_parallel_workers_per_gather = 0;
+
+-- Trivial asymmetric JOIN of partitioned table with a relation
+EXPLAIN (COSTS OFF)
+SELECT *
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+
+-- The same, but appended with UNION ALL
+EXPLAIN (COSTS OFF)
+SELECT * FROM (
+ (SELECT * FROM prt5_p0)
+ UNION ALL
+ (SELECT * FROM prt5_p1)
+ UNION ALL
+ (SELECT * FROM prt5_p2)
+ ) AS sq1
+JOIN t5_1 ON a = aid AND alabel like '%abc%';
+
+-- Don't allow asymmetric JOIN of two partitioned tables.
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM prt5 JOIN prt6 ON a = aid AND alabel like '%abc%';
+SELECT count(*) FROM prt5 JOIN prt6 ON a = aid AND alabel like '%abc%';
+
+-- Check asymmetric JOIN with Subquery
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM prt5 JOIN (
+ SELECT * FROM prt6 LIMIT 1000
+) AS sq1 ON a = aid AND alabel like '%abc%';
+SELECT count(*) FROM prt5 JOIN (SELECT * FROM prt6 LIMIT 1000) AS sq1
+ ON a = aid AND alabel like '%abc%';
+
+-- Asymmetric JOIN of two plane tables and one partitioned
+EXPLAIN (COSTS OFF)
+SELECT count(*)
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+SELECT count(*)
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+
+-- unable to extract non-partitioned right relation
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 RIGHT JOIN t5_1 ON a = aid AND alabel like '%abc%';
+-- left side can be extracted, but no cost benefit
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 LEFT JOIN t5_1 ON a = aid AND alabel like '%abc%';
+
+-- validation of the results with/without asymmetric partition-wise join
+SELECT * INTO pg_temp.result01a
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+
+SELECT * INTO pg_temp.result02a
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+
+SET enable_partitionwise_join = off;
+
+SELECT * INTO pg_temp.result01b
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+
+SELECT * INTO pg_temp.result02b
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+
+SELECT * FROM pg_temp.result01a EXCEPT SELECT * FROM pg_temp.result01b;
+SELECT * FROM pg_temp.result02a EXCEPT SELECT * FROM pg_temp.result02b;
+
+RESET max_parallel_workers_per_gather;
+SET enable_partitionwise_join = true;
+
+-- Parameterized path examples.
+CREATE TABLE prta (id integer, payload text) PARTITION BY HASH (id);
+CREATE TABLE prta1 PARTITION OF prta FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE prta2 PARTITION OF prta FOR VALUES WITH (modulus 2, remainder 1);
+CREATE UNIQUE INDEX ON prta1 (id);
+CREATE UNIQUE INDEX ON prta2 (id);
+INSERT INTO prta (id, payload)
+ (SELECT *, ('abc' || id)::text AS payload
+ FROM generate_series(1,1) AS id);
+
+CREATE TABLE prtb (id integer, payload text) PARTITION BY HASH (id);
+CREATE TABLE prtb1 PARTITION OF prtb FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE prtb2 PARTITION OF prtb FOR VALUES WITH (modulus 2, remainder 1);
+CREATE UNIQUE INDEX ON prtb1 (id);
+CREATE UNIQUE INDEX ON prtb2 (id);
+INSERT INTO prtb (id, payload)
+ (SELECT *, ('def' || id)::text AS payload
+ FROM generate_series(1,1000) AS id);
+
+CREATE TABLE e (id integer, payload text) PARTITION BY HASH (id);
+CREATE TABLE e1 PARTITION OF e FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE e2 PARTITION OF e FOR VALUES WITH (modulus 2, remainder 1);
+INSERT INTO e (id, payload)
+ (SELECT *, ('ghi' || id)::text AS payload
+ FROM generate_series(1,1000) AS id);
+CREATE UNIQUE INDEX ON e1 (id);
+CREATE UNIQUE INDEX ON e2 (id);
+
+ANALYZE prta,prtb,e;
+
+EXPLAIN (COSTS OFF)
+SELECT * FROM prta,prtb WHERE prta.id=prtb.id;
+
+EXPLAIN (COSTS OFF)
+SELECT * FROM prta,prtb,e WHERE prta.id=prtb.id AND prta.id=e.id;
+
-- semi join
EXPLAIN (COSTS OFF)
SELECT t1.* FROM prt1_adv t1 WHERE EXISTS (SELECT 1 FROM prt2_adv t2 WHERE t1.a = t2.b) AND t1.b = 0 ORDER BY t1.a;
--
2.31.1
Attachments:
[text/plain] v17-0001-Asymmetric-partitionwise-join.patch (35.0K, ../../[email protected]/2-v17-0001-Asymmetric-partitionwise-join.patch)
download | inline diff:
From 7b05d8535b11ed7b5c43ed591d9aeea955f2c2ed Mon Sep 17 00:00:00 2001
From: Andrey Lepikhov <[email protected]>
Date: Fri, 2 Apr 2021 11:02:20 +0500
Subject: [PATCH] Asymmetric partitionwise join.
Teach optimizer to consider partitionwise join of non-partitioned
table with each partition of partitioned table.
Disallow asymmetric machinery for joining of two partitioned (or appended)
relations because it could cause huge consumption of CPU and memory
during reparameterization of NestLoop path.
---
src/backend/optimizer/path/joinpath.c | 9 +
src/backend/optimizer/path/joinrels.c | 187 +++++++++++
src/backend/optimizer/plan/setrefs.c | 17 +-
src/backend/optimizer/util/appendinfo.c | 37 ++-
src/backend/optimizer/util/pathnode.c | 9 +-
src/backend/optimizer/util/relnode.c | 19 +-
src/include/optimizer/paths.h | 7 +-
src/test/regress/expected/partition_join.out | 332 +++++++++++++++++++
src/test/regress/sql/partition_join.sql | 133 ++++++++
9 files changed, 720 insertions(+), 30 deletions(-)
diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c
index 6407ede12a..32618ebbd5 100644
--- a/src/backend/optimizer/path/joinpath.c
+++ b/src/backend/optimizer/path/joinpath.c
@@ -335,6 +335,15 @@ add_paths_to_joinrel(PlannerInfo *root,
if (set_join_pathlist_hook)
set_join_pathlist_hook(root, joinrel, outerrel, innerrel,
jointype, &extra);
+
+ /*
+ * 7. If outer relation is delivered from partition-tables, consider
+ * distributing inner relation to every partition-leaf prior to
+ * append these leafs.
+ */
+ try_asymmetric_partitionwise_join(root, joinrel,
+ outerrel, innerrel,
+ jointype, &extra);
}
/*
diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c
index 0dbe2ac726..c8db6f0969 100644
--- a/src/backend/optimizer/path/joinrels.c
+++ b/src/backend/optimizer/path/joinrels.c
@@ -16,6 +16,7 @@
#include "miscadmin.h"
#include "optimizer/appendinfo.h"
+#include "optimizer/cost.h"
#include "optimizer/joininfo.h"
#include "optimizer/pathnode.h"
#include "optimizer/paths.h"
@@ -1551,6 +1552,192 @@ try_partitionwise_join(PlannerInfo *root, RelOptInfo *rel1, RelOptInfo *rel2,
}
}
+/*
+ * Build RelOptInfo on JOIN of each partition of the outer relation and the inner
+ * relation. Return List of such RelOptInfo's. Return NIL, if at least one of
+ * these JOINs is impossible to build.
+ */
+static List *
+extract_asymmetric_partitionwise_subjoin(PlannerInfo *root,
+ RelOptInfo *joinrel,
+ AppendPath *append_path,
+ RelOptInfo *inner_rel,
+ JoinType jointype,
+ JoinPathExtraData *extra)
+{
+ List *result = NIL;
+ ListCell *lc;
+
+ foreach (lc, append_path->subpaths)
+ {
+ Path *child_path = lfirst(lc);
+ RelOptInfo *child_rel = child_path->parent;
+ Relids child_joinrelids;
+ Relids parent_relids;
+ RelOptInfo *child_joinrel;
+ SpecialJoinInfo *child_sjinfo;
+ List *child_restrictlist;
+
+ child_joinrelids = bms_union(child_rel->relids, inner_rel->relids);
+ parent_relids = bms_union(append_path->path.parent->relids,
+ inner_rel->relids);
+
+ child_sjinfo = build_child_join_sjinfo(root, extra->sjinfo,
+ child_rel->relids,
+ inner_rel->relids);
+ child_restrictlist = (List *)
+ adjust_appendrel_attrs_multilevel(root, (Node *)extra->restrictlist,
+ child_joinrelids, parent_relids);
+
+ child_joinrel = find_join_rel(root, child_joinrelids);
+ if (!child_joinrel)
+ child_joinrel = build_child_join_rel(root,
+ child_rel,
+ inner_rel,
+ joinrel,
+ child_restrictlist,
+ child_sjinfo,
+ jointype);
+ else
+ {
+ /*
+ * The join relation already exists. For example, it could happen if
+ * we join two plane tables with partitioned table(s).
+ * Populating this join with additional paths could push out some
+ * previously added paths which could be pointed in a subplans list
+ * of an higher level append.
+ * Of course, we could save such paths before generating new. But it
+ * can increase too much the number of paths in complex queries. It
+ * can be a task for future work.
+ */
+ return NIL;
+ }
+
+ populate_joinrel_with_paths(root,
+ child_rel,
+ inner_rel,
+ child_joinrel,
+ child_sjinfo,
+ child_restrictlist);
+
+ /* Give up if asymmetric partition-wise join is not available */
+ if (child_joinrel->pathlist == NIL)
+ return NIL;
+
+ set_cheapest(child_joinrel);
+ result = lappend(result, child_joinrel);
+ }
+ return result;
+}
+
+static bool
+is_asymmetric_join_feasible(PlannerInfo *root,
+ RelOptInfo *outer_rel,
+ RelOptInfo *inner_rel,
+ JoinType jointype)
+{
+ ListCell *lc;
+
+ if (jointype != JOIN_INNER && jointype != JOIN_LEFT)
+ return false;
+
+ if (IS_OTHER_REL(outer_rel) || IS_OTHER_REL(inner_rel))
+ return false;
+
+ /* Disallow recursive usage of asymmertic join machinery */
+ if (root->join_rel_level == NULL)
+ return false;
+
+ /*
+ * Don't allow asymmetric JOIN of two append subplans.
+ * In the case of a parameterized NL join, a reparameterization procedure
+ * will lead to large memory allocations and a CPU consumption:
+ * each reparameterization will induce subpath duplication, creating new
+ * ParamPathInfo instance and increasing of ppilist up to number of
+ * partitions in the inner. Also, if we have many partitions, each bitmapset
+ * variable will be large and many leaks of such variable (caused by relid
+ * replacement) will highly increase memory consumption.
+ * So, we deny such paths for now.
+ */
+ foreach(lc, inner_rel->pathlist)
+ {
+ if (IsA(lfirst(lc), AppendPath))
+ return false;
+ }
+
+ return true;
+}
+
+void
+try_asymmetric_partitionwise_join(PlannerInfo *root,
+ RelOptInfo *joinrel,
+ RelOptInfo *outer_rel,
+ RelOptInfo *inner_rel,
+ JoinType jointype,
+ JoinPathExtraData *extra)
+{
+ ListCell *lc;
+
+ /*
+ * Try this kind of paths if we allow complex partitionwise joins and we know
+ * we can build this join safely.
+ */
+ if (!enable_partitionwise_join ||
+ !is_asymmetric_join_feasible(root, outer_rel, inner_rel, jointype))
+ return;
+
+ foreach (lc, outer_rel->pathlist)
+ {
+ AppendPath *append_path = lfirst(lc);
+
+ /*
+ * We assume this pathlist keeps at least one AppendPath that
+ * represents partitioned table-scan, symmetric or asymmetric
+ * partition-wise join. Asymmetric join isn't needed if the append node
+ * has only one child.
+ */
+ if (IsA(append_path, AppendPath) &&
+ list_length(append_path->subpaths) > 1)
+ {
+ List **join_rel_level_saved;
+ List *live_childrels = NIL;
+
+ join_rel_level_saved = root->join_rel_level;
+ PG_TRY();
+ {
+ /* temporary disables "dynamic programming" algorithm */
+ root->join_rel_level = NULL;
+
+ live_childrels =
+ extract_asymmetric_partitionwise_subjoin(root,
+ joinrel,
+ append_path,
+ inner_rel,
+ jointype,
+ extra);
+ }
+ PG_FINALLY();
+ {
+ root->join_rel_level = join_rel_level_saved;
+ }
+ PG_END_TRY();
+
+ if (live_childrels != NIL)
+ {
+ /*
+ * Add new append relation. We must choose cheapest paths after
+ * this operation because the pathlist possibly contains
+ * joinrels and appendrels that can be suboptimal.
+ */
+ add_paths_to_append_rel(root, joinrel, live_childrels);
+ set_cheapest(joinrel);
+ }
+
+ break;
+ }
+ }
+}
+
/*
* Construct the SpecialJoinInfo for a child-join by translating
* SpecialJoinInfo for the join between parents. left_relids and right_relids
diff --git a/src/backend/optimizer/plan/setrefs.c b/src/backend/optimizer/plan/setrefs.c
index 26f6872b4b..6ffa68efba 100644
--- a/src/backend/optimizer/plan/setrefs.c
+++ b/src/backend/optimizer/plan/setrefs.c
@@ -281,24 +281,29 @@ set_plan_references(PlannerInfo *root, Plan *plan)
/*
* Adjust RT indexes of AppendRelInfos and add to final appendrels list.
- * We assume the AppendRelInfos were built during planning and don't need
- * to be copied.
+ * The AppendRelInfos are copied, because as a part of a subplan they could
+ * be visited many times in the case of asymmetric join.
*/
foreach(lc, root->append_rel_list)
{
AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc);
+ AppendRelInfo *newappinfo;
+
+ /* flat copy is enough since all valuable fields are scalars */
+ newappinfo = (AppendRelInfo *) palloc(sizeof(AppendRelInfo));
+ memcpy(newappinfo, appinfo, sizeof(AppendRelInfo));
/* adjust RT indexes */
- appinfo->parent_relid += rtoffset;
- appinfo->child_relid += rtoffset;
+ newappinfo->parent_relid += rtoffset;
+ newappinfo->child_relid += rtoffset;
/*
* Rather than adjust the translated_vars entries, just drop 'em.
* Neither the executor nor EXPLAIN currently need that data.
*/
- appinfo->translated_vars = NIL;
+ newappinfo->translated_vars = NIL;
- glob->appendRelations = lappend(glob->appendRelations, appinfo);
+ glob->appendRelations = lappend(glob->appendRelations, newappinfo);
}
/* Now fix the Plan tree */
diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c
index af46f581ac..ec8dee99b0 100644
--- a/src/backend/optimizer/util/appendinfo.c
+++ b/src/backend/optimizer/util/appendinfo.c
@@ -202,7 +202,9 @@ adjust_appendrel_attrs(PlannerInfo *root, Node *node, int nappinfos,
context.appinfos = appinfos;
/* If there's nothing to adjust, don't call this function. */
- Assert(nappinfos >= 1 && appinfos != NULL);
+ /* If there's nothing to adjust, just return a duplication */
+ if (nappinfos == 0)
+ return copyObject(node);
/* Should never be translating a Query tree. */
Assert(node == NULL || !IsA(node, Query));
@@ -489,12 +491,10 @@ adjust_appendrel_attrs_multilevel(PlannerInfo *root, Node *node,
Relids child_relids,
Relids top_parent_relids)
{
- AppendRelInfo **appinfos;
- Bitmapset *parent_relids = NULL;
- int nappinfos;
- int cnt;
-
- Assert(bms_num_members(child_relids) == bms_num_members(top_parent_relids));
+ AppendRelInfo **appinfos;
+ Relids parent_relids = NULL;
+ int nappinfos;
+ int cnt;
appinfos = find_appinfos_by_relids(root, child_relids, &nappinfos);
@@ -506,8 +506,13 @@ adjust_appendrel_attrs_multilevel(PlannerInfo *root, Node *node,
parent_relids = bms_add_member(parent_relids, appinfo->parent_relid);
}
- /* Recurse if immediate parent is not the top parent. */
- if (!bms_equal(parent_relids, top_parent_relids))
+ /*
+ * Recurse if immediate parent is not the top parent. Keep in mind that in a
+ * case of asymmetric JOIN top_parent_relids can contain relids which aren't
+ * part of an append node.
+ */
+ if (!bms_equal(parent_relids, top_parent_relids) &&
+ !bms_is_subset(parent_relids, top_parent_relids))
node = adjust_appendrel_attrs_multilevel(root, node, parent_relids,
top_parent_relids);
@@ -515,6 +520,7 @@ adjust_appendrel_attrs_multilevel(PlannerInfo *root, Node *node,
node = adjust_appendrel_attrs(root, node, nappinfos, appinfos);
pfree(appinfos);
+ pfree(parent_relids);
return node;
}
@@ -565,6 +571,7 @@ adjust_child_relids_multilevel(PlannerInfo *root, Relids relids,
AppendRelInfo **appinfos;
int nappinfos;
Relids parent_relids = NULL;
+ Relids normal_relids = NULL;
Relids result;
Relids tmp_result = NULL;
int cnt;
@@ -579,12 +586,17 @@ adjust_child_relids_multilevel(PlannerInfo *root, Relids relids,
appinfos = find_appinfos_by_relids(root, child_relids, &nappinfos);
/* Construct relids set for the immediate parent of the given child. */
+ normal_relids = bms_copy(child_relids);
for (cnt = 0; cnt < nappinfos; cnt++)
{
AppendRelInfo *appinfo = appinfos[cnt];
parent_relids = bms_add_member(parent_relids, appinfo->parent_relid);
+ normal_relids = bms_del_member(normal_relids, appinfo->child_relid);
}
+ parent_relids = bms_union(parent_relids, normal_relids);
+ if (normal_relids)
+ bms_free(normal_relids);
/* Recurse if immediate parent is not the top parent. */
if (!bms_equal(parent_relids, top_parent_relids))
@@ -715,11 +727,11 @@ AppendRelInfo **
find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos)
{
AppendRelInfo **appinfos;
+ int nrooms = bms_num_members(relids);
int cnt = 0;
int i;
- *nappinfos = bms_num_members(relids);
- appinfos = (AppendRelInfo **) palloc(sizeof(AppendRelInfo *) * *nappinfos);
+ appinfos = (AppendRelInfo **) palloc(sizeof(AppendRelInfo *) * nrooms);
i = -1;
while ((i = bms_next_member(relids, i)) >= 0)
@@ -727,10 +739,11 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos)
AppendRelInfo *appinfo = root->append_rel_array[i];
if (!appinfo)
- elog(ERROR, "child rel %d not found in append_rel_array", i);
+ continue;
appinfos[cnt++] = appinfo;
}
+ *nappinfos = cnt;
return appinfos;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index 0c94cbe767..e4828b3647 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -4224,7 +4224,14 @@ do { \
MemoryContextSwitchTo(oldcontext);
}
- bms_free(required_outer);
+
+ /*
+ * If adjust_child_relids_multilevel don't do replacements it returns
+ * the original set, not a copy. It is possible in the case of asymmetric
+ * JOIN and child_rel->relids contains relids only of plane relations.
+ */
+ if (required_outer != old_ppi->ppi_req_outer)
+ bms_free(required_outer);
new_path->param_info = new_ppi;
diff --git a/src/backend/optimizer/util/relnode.c b/src/backend/optimizer/util/relnode.c
index e105a4d5f1..bb15fb2716 100644
--- a/src/backend/optimizer/util/relnode.c
+++ b/src/backend/optimizer/util/relnode.c
@@ -790,11 +790,8 @@ build_child_join_rel(PlannerInfo *root, RelOptInfo *outer_rel,
AppendRelInfo **appinfos;
int nappinfos;
- /* Only joins between "other" relations land here. */
- Assert(IS_OTHER_REL(outer_rel) && IS_OTHER_REL(inner_rel));
-
- /* The parent joinrel should have consider_partitionwise_join set. */
- Assert(parent_joinrel->consider_partitionwise_join);
+ /* Either of relations must be "other" relation at least. */
+ Assert(IS_OTHER_REL(outer_rel) || IS_OTHER_REL(inner_rel));
joinrel->reloptkind = RELOPT_OTHER_JOINREL;
joinrel->relids = bms_union(outer_rel->relids, inner_rel->relids);
@@ -851,8 +848,11 @@ build_child_join_rel(PlannerInfo *root, RelOptInfo *outer_rel,
joinrel->partexprs = NULL;
joinrel->nullable_partexprs = NULL;
- joinrel->top_parent_relids = bms_union(outer_rel->top_parent_relids,
- inner_rel->top_parent_relids);
+ joinrel->top_parent_relids =
+ bms_union(IS_OTHER_REL(outer_rel) ?
+ outer_rel->top_parent_relids : outer_rel->relids,
+ IS_OTHER_REL(inner_rel) ?
+ inner_rel->top_parent_relids : inner_rel->relids);
/* Compute information relevant to foreign relations. */
set_foreign_rel_properties(joinrel, outer_rel, inner_rel);
@@ -2033,9 +2033,8 @@ build_child_join_reltarget(PlannerInfo *root,
{
/* Build the targetlist */
childrel->reltarget->exprs = (List *)
- adjust_appendrel_attrs(root,
- (Node *) parentrel->reltarget->exprs,
- nappinfos, appinfos);
+ adjust_appendrel_attrs_multilevel(root, (Node *)parentrel->reltarget->exprs,
+ childrel->relids, parentrel->relids);
/* Set the cost and width fields */
childrel->reltarget->cost.startup = parentrel->reltarget->cost.startup;
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index f1d111063c..a0106dc539 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -109,7 +109,12 @@ extern bool have_join_order_restriction(PlannerInfo *root,
extern bool have_dangerous_phv(PlannerInfo *root,
Relids outer_relids, Relids inner_params);
extern void mark_dummy_rel(RelOptInfo *rel);
-
+extern void try_asymmetric_partitionwise_join(PlannerInfo *root,
+ RelOptInfo *joinrel,
+ RelOptInfo *outer_rel,
+ RelOptInfo *inner_rel,
+ JoinType jointype,
+ JoinPathExtraData *extra);
/*
* equivclass.c
* routines for managing EquivalenceClasses
diff --git a/src/test/regress/expected/partition_join.out b/src/test/regress/expected/partition_join.out
index 27f7525b3e..753b3de47f 100644
--- a/src/test/regress/expected/partition_join.out
+++ b/src/test/regress/expected/partition_join.out
@@ -2320,6 +2320,338 @@ SELECT t1.a, t1.c, t2.b, t2.c FROM prt1_adv t1 INNER JOIN prt2_adv t2 ON (t1.a =
375 | 0375 | 375 | 0375
(8 rows)
+--
+-- For asymmetric partition-wise join
+--
+CREATE TABLE prt5 (hkey int, a int, b int) PARTITION BY HASH(hkey);
+CREATE TABLE prt5_p0 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 0);
+CREATE TABLE prt5_p1 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 1);
+CREATE TABLE prt5_p2 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 2);
+CREATE TABLE prt6 (aid int, alabel text) PARTITION BY HASH(aid);
+CREATE TABLE prt6_p0 PARTITION OF prt6
+ FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE prt6_p1 PARTITION OF prt6
+ FOR VALUES WITH (modulus 2, remainder 1);
+CREATE TABLE t5_1 (aid int, alabel text);
+CREATE TABLE t5_2 (bid int, blabel text);
+INSERT INTO prt5 (SELECT x, (x % 1000)::int,
+ ((x+1) % 1000)::int
+ FROM generate_series(1,1000000) x);
+INSERT INTO t5_1 (SELECT x, md5(x::text) FROM generate_series(-200, 800) x);
+INSERT INTO t5_2 (SELECT x, md5(x::text) FROM generate_series(-200, 800) x);
+INSERT INTO prt6 (SELECT * FROM t5_1);
+VACUUM ANALYZE prt5,prt6,t5_1,t5_2;
+SET max_parallel_workers_per_gather = 0;
+-- Trivial asymmetric JOIN of partitioned table with a relation
+EXPLAIN (COSTS OFF)
+SELECT *
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+-------------------------------------------------------
+ Append
+ -> Hash Join
+ Hash Cond: (prt5_1.a = t5_1.aid)
+ -> Seq Scan on prt5_p0 prt5_1
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_2.a = t5_1.aid)
+ -> Seq Scan on prt5_p1 prt5_2
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_3.a = t5_1.aid)
+ -> Seq Scan on prt5_p2 prt5_3
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+(19 rows)
+
+-- The same, but appended with UNION ALL
+EXPLAIN (COSTS OFF)
+SELECT * FROM (
+ (SELECT * FROM prt5_p0)
+ UNION ALL
+ (SELECT * FROM prt5_p1)
+ UNION ALL
+ (SELECT * FROM prt5_p2)
+ ) AS sq1
+JOIN t5_1 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+-------------------------------------------------------
+ Append
+ -> Hash Join
+ Hash Cond: (prt5_p0.a = t5_1.aid)
+ -> Seq Scan on prt5_p0
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_p1.a = t5_1.aid)
+ -> Seq Scan on prt5_p1
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_p2.a = t5_1.aid)
+ -> Seq Scan on prt5_p2
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+(19 rows)
+
+-- Don't allow asymmetric JOIN of two partitioned tables.
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM prt5 JOIN prt6 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+-------------------------------------------------------------
+ Aggregate
+ -> Hash Join
+ Hash Cond: (prt5.a = prt6.aid)
+ -> Append
+ -> Seq Scan on prt5_p0 prt5_1
+ -> Seq Scan on prt5_p1 prt5_2
+ -> Seq Scan on prt5_p2 prt5_3
+ -> Hash
+ -> Append
+ -> Seq Scan on prt6_p0 prt6_1
+ Filter: (alabel ~~ '%abc%'::text)
+ -> Seq Scan on prt6_p1 prt6_2
+ Filter: (alabel ~~ '%abc%'::text)
+(13 rows)
+
+SELECT count(*) FROM prt5 JOIN prt6 ON a = aid AND alabel like '%abc%';
+ count
+-------
+ 4000
+(1 row)
+
+-- Check asymmetric JOIN with Subquery
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM prt5 JOIN (
+ SELECT * FROM prt6 LIMIT 1000
+) AS sq1 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+----------------------------------------------------------------------
+ Aggregate
+ -> Append
+ -> Hash Join
+ Hash Cond: (prt5_1.a = sq1.aid)
+ -> Seq Scan on prt5_p0 prt5_1
+ -> Hash
+ -> Subquery Scan on sq1
+ Filter: (sq1.alabel ~~ '%abc%'::text)
+ -> Limit
+ -> Append
+ -> Seq Scan on prt6_p0 prt6_1
+ -> Seq Scan on prt6_p1 prt6_2
+ -> Hash Join
+ Hash Cond: (prt5_2.a = sq1.aid)
+ -> Seq Scan on prt5_p1 prt5_2
+ -> Hash
+ -> Subquery Scan on sq1
+ Filter: (sq1.alabel ~~ '%abc%'::text)
+ -> Limit
+ -> Append
+ -> Seq Scan on prt6_p0 prt6_4
+ -> Seq Scan on prt6_p1 prt6_5
+ -> Hash Join
+ Hash Cond: (prt5_3.a = sq1.aid)
+ -> Seq Scan on prt5_p2 prt5_3
+ -> Hash
+ -> Subquery Scan on sq1
+ Filter: (sq1.alabel ~~ '%abc%'::text)
+ -> Limit
+ -> Append
+ -> Seq Scan on prt6_p0 prt6_7
+ -> Seq Scan on prt6_p1 prt6_8
+(32 rows)
+
+SELECT count(*) FROM prt5 JOIN (SELECT * FROM prt6 LIMIT 1000) AS sq1
+ ON a = aid AND alabel like '%abc%';
+ count
+-------
+ 4000
+(1 row)
+
+-- Asymmetric JOIN of two plane tables and one partitioned
+EXPLAIN (COSTS OFF)
+SELECT count(*)
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+ QUERY PLAN
+------------------------------------------------------------------
+ Aggregate
+ -> Append
+ -> Hash Join
+ Hash Cond: (prt5_1.a = t5_1.aid)
+ -> Hash Join
+ Hash Cond: (prt5_1.b = t5_2.bid)
+ -> Seq Scan on prt5_p0 prt5_1
+ -> Hash
+ -> Seq Scan on t5_2
+ Filter: (blabel ~~ '%cd%'::text)
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%ab%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_2.a = t5_1.aid)
+ -> Hash Join
+ Hash Cond: (prt5_2.b = t5_2.bid)
+ -> Seq Scan on prt5_p1 prt5_2
+ -> Hash
+ -> Seq Scan on t5_2
+ Filter: (blabel ~~ '%cd%'::text)
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%ab%'::text)
+ -> Hash Join
+ Hash Cond: (prt5_3.a = t5_1.aid)
+ -> Hash Join
+ Hash Cond: (prt5_3.b = t5_2.bid)
+ -> Seq Scan on prt5_p2 prt5_3
+ -> Hash
+ -> Seq Scan on t5_2
+ Filter: (blabel ~~ '%cd%'::text)
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%ab%'::text)
+(35 rows)
+
+SELECT count(*)
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+ count
+-------
+ 10000
+(1 row)
+
+-- unable to extract non-partitioned right relation
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 RIGHT JOIN t5_1 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+-----------------------------------------------
+ Hash Right Join
+ Hash Cond: (prt5.a = t5_1.aid)
+ Join Filter: (t5_1.alabel ~~ '%abc%'::text)
+ -> Append
+ -> Seq Scan on prt5_p0 prt5_1
+ -> Seq Scan on prt5_p1 prt5_2
+ -> Seq Scan on prt5_p2 prt5_3
+ -> Hash
+ -> Seq Scan on t5_1
+(9 rows)
+
+-- left side can be extracted, but no cost benefit
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 LEFT JOIN t5_1 ON a = aid AND alabel like '%abc%';
+ QUERY PLAN
+-------------------------------------------------
+ Hash Left Join
+ Hash Cond: (prt5.a = t5_1.aid)
+ -> Append
+ -> Seq Scan on prt5_p0 prt5_1
+ -> Seq Scan on prt5_p1 prt5_2
+ -> Seq Scan on prt5_p2 prt5_3
+ -> Hash
+ -> Seq Scan on t5_1
+ Filter: (alabel ~~ '%abc%'::text)
+(9 rows)
+
+SET enable_partitionwise_join = on;
+-- validation of the results with/without asymmetric partition-wise join
+SELECT * INTO pg_temp.result01a
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+SELECT * INTO pg_temp.result02a
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+SET enable_partitionwise_join = off;
+SELECT * INTO pg_temp.result01b
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+SELECT * INTO pg_temp.result02b
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+SELECT * FROM pg_temp.result01a EXCEPT SELECT * FROM pg_temp.result01b;
+ hkey | a | b | aid | alabel
+------+---+---+-----+--------
+(0 rows)
+
+SELECT * FROM pg_temp.result02a EXCEPT SELECT * FROM pg_temp.result02b;
+ hkey | a | b | aid | alabel | bid | blabel
+------+---+---+-----+--------+-----+--------
+(0 rows)
+
+RESET max_parallel_workers_per_gather;
+SET enable_partitionwise_join = true;
+-- Parameterized path examples.
+CREATE TABLE prta (id integer, payload text) PARTITION BY HASH (id);
+CREATE TABLE prta1 PARTITION OF prta FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE prta2 PARTITION OF prta FOR VALUES WITH (modulus 2, remainder 1);
+CREATE UNIQUE INDEX ON prta1 (id);
+CREATE UNIQUE INDEX ON prta2 (id);
+INSERT INTO prta (id, payload)
+ (SELECT *, ('abc' || id)::text AS payload
+ FROM generate_series(1,1) AS id);
+CREATE TABLE prtb (id integer, payload text) PARTITION BY HASH (id);
+CREATE TABLE prtb1 PARTITION OF prtb FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE prtb2 PARTITION OF prtb FOR VALUES WITH (modulus 2, remainder 1);
+CREATE UNIQUE INDEX ON prtb1 (id);
+CREATE UNIQUE INDEX ON prtb2 (id);
+INSERT INTO prtb (id, payload)
+ (SELECT *, ('def' || id)::text AS payload
+ FROM generate_series(1,1000) AS id);
+CREATE TABLE e (id integer, payload text) PARTITION BY HASH (id);
+CREATE TABLE e1 PARTITION OF e FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE e2 PARTITION OF e FOR VALUES WITH (modulus 2, remainder 1);
+INSERT INTO e (id, payload)
+ (SELECT *, ('ghi' || id)::text AS payload
+ FROM generate_series(1,1000) AS id);
+CREATE UNIQUE INDEX ON e1 (id);
+CREATE UNIQUE INDEX ON e2 (id);
+ANALYZE prta,prtb,e;
+EXPLAIN (COSTS OFF)
+SELECT * FROM prta,prtb WHERE prta.id=prtb.id;
+ QUERY PLAN
+-----------------------------------------------------------
+ Append
+ -> Nested Loop
+ -> Seq Scan on prta1 prta_1
+ -> Index Scan using prtb1_id_idx on prtb1 prtb_1
+ Index Cond: (id = prta_1.id)
+ -> Nested Loop
+ -> Seq Scan on prta2 prta_2
+ -> Index Scan using prtb2_id_idx on prtb2 prtb_2
+ Index Cond: (id = prta_2.id)
+(9 rows)
+
+EXPLAIN (COSTS OFF)
+SELECT * FROM prta,prtb,e WHERE prta.id=prtb.id AND prta.id=e.id;
+ QUERY PLAN
+-----------------------------------------------------------------
+ Append
+ -> Nested Loop
+ Join Filter: (prta_1.id = e_1.id)
+ -> Nested Loop
+ -> Seq Scan on prta1 prta_1
+ -> Index Scan using prtb1_id_idx on prtb1 prtb_1
+ Index Cond: (id = prta_1.id)
+ -> Index Scan using e1_id_idx on e1 e_1
+ Index Cond: (id = prtb_1.id)
+ -> Nested Loop
+ Join Filter: (prta_2.id = e_2.id)
+ -> Nested Loop
+ -> Seq Scan on prta2 prta_2
+ -> Index Scan using prtb2_id_idx on prtb2 prtb_2
+ Index Cond: (id = prta_2.id)
+ -> Index Scan using e2_id_idx on e2 e_2
+ Index Cond: (id = prtb_2.id)
+(17 rows)
+
-- semi join
EXPLAIN (COSTS OFF)
SELECT t1.* FROM prt1_adv t1 WHERE EXISTS (SELECT 1 FROM prt2_adv t2 WHERE t1.a = t2.b) AND t1.b = 0 ORDER BY t1.a;
diff --git a/src/test/regress/sql/partition_join.sql b/src/test/regress/sql/partition_join.sql
index d97b5b69ff..aca7f885a3 100644
--- a/src/test/regress/sql/partition_join.sql
+++ b/src/test/regress/sql/partition_join.sql
@@ -536,6 +536,139 @@ EXPLAIN (COSTS OFF)
SELECT t1.a, t1.c, t2.b, t2.c FROM prt1_adv t1 INNER JOIN prt2_adv t2 ON (t1.a = t2.b) WHERE t1.b = 0 ORDER BY t1.a, t2.b;
SELECT t1.a, t1.c, t2.b, t2.c FROM prt1_adv t1 INNER JOIN prt2_adv t2 ON (t1.a = t2.b) WHERE t1.b = 0 ORDER BY t1.a, t2.b;
+--
+-- For asymmetric partition-wise join
+--
+CREATE TABLE prt5 (hkey int, a int, b int) PARTITION BY HASH(hkey);
+CREATE TABLE prt5_p0 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 0);
+CREATE TABLE prt5_p1 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 1);
+CREATE TABLE prt5_p2 PARTITION OF prt5
+ FOR VALUES WITH (modulus 3, remainder 2);
+CREATE TABLE prt6 (aid int, alabel text) PARTITION BY HASH(aid);
+CREATE TABLE prt6_p0 PARTITION OF prt6
+ FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE prt6_p1 PARTITION OF prt6
+ FOR VALUES WITH (modulus 2, remainder 1);
+CREATE TABLE t5_1 (aid int, alabel text);
+CREATE TABLE t5_2 (bid int, blabel text);
+
+INSERT INTO prt5 (SELECT x, (x % 1000)::int,
+ ((x+1) % 1000)::int
+ FROM generate_series(1,1000000) x);
+INSERT INTO t5_1 (SELECT x, md5(x::text) FROM generate_series(-200, 800) x);
+INSERT INTO t5_2 (SELECT x, md5(x::text) FROM generate_series(-200, 800) x);
+INSERT INTO prt6 (SELECT * FROM t5_1);
+
+VACUUM ANALYZE prt5,prt6,t5_1,t5_2;
+
+SET max_parallel_workers_per_gather = 0;
+
+-- Trivial asymmetric JOIN of partitioned table with a relation
+EXPLAIN (COSTS OFF)
+SELECT *
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+
+-- The same, but appended with UNION ALL
+EXPLAIN (COSTS OFF)
+SELECT * FROM (
+ (SELECT * FROM prt5_p0)
+ UNION ALL
+ (SELECT * FROM prt5_p1)
+ UNION ALL
+ (SELECT * FROM prt5_p2)
+ ) AS sq1
+JOIN t5_1 ON a = aid AND alabel like '%abc%';
+
+-- Don't allow asymmetric JOIN of two partitioned tables.
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM prt5 JOIN prt6 ON a = aid AND alabel like '%abc%';
+SELECT count(*) FROM prt5 JOIN prt6 ON a = aid AND alabel like '%abc%';
+
+-- Check asymmetric JOIN with Subquery
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM prt5 JOIN (
+ SELECT * FROM prt6 LIMIT 1000
+) AS sq1 ON a = aid AND alabel like '%abc%';
+SELECT count(*) FROM prt5 JOIN (SELECT * FROM prt6 LIMIT 1000) AS sq1
+ ON a = aid AND alabel like '%abc%';
+
+-- Asymmetric JOIN of two plane tables and one partitioned
+EXPLAIN (COSTS OFF)
+SELECT count(*)
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+SELECT count(*)
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+
+-- unable to extract non-partitioned right relation
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 RIGHT JOIN t5_1 ON a = aid AND alabel like '%abc%';
+-- left side can be extracted, but no cost benefit
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt5 LEFT JOIN t5_1 ON a = aid AND alabel like '%abc%';
+
+-- validation of the results with/without asymmetric partition-wise join
+SELECT * INTO pg_temp.result01a
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+
+SELECT * INTO pg_temp.result02a
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+
+SET enable_partitionwise_join = off;
+
+SELECT * INTO pg_temp.result01b
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%abc%';
+
+SELECT * INTO pg_temp.result02b
+ FROM prt5 JOIN t5_1 ON a = aid AND alabel like '%ab%'
+ JOIN t5_2 ON b = bid AND blabel like '%cd%';
+
+SELECT * FROM pg_temp.result01a EXCEPT SELECT * FROM pg_temp.result01b;
+SELECT * FROM pg_temp.result02a EXCEPT SELECT * FROM pg_temp.result02b;
+
+RESET max_parallel_workers_per_gather;
+SET enable_partitionwise_join = true;
+
+-- Parameterized path examples.
+CREATE TABLE prta (id integer, payload text) PARTITION BY HASH (id);
+CREATE TABLE prta1 PARTITION OF prta FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE prta2 PARTITION OF prta FOR VALUES WITH (modulus 2, remainder 1);
+CREATE UNIQUE INDEX ON prta1 (id);
+CREATE UNIQUE INDEX ON prta2 (id);
+INSERT INTO prta (id, payload)
+ (SELECT *, ('abc' || id)::text AS payload
+ FROM generate_series(1,1) AS id);
+
+CREATE TABLE prtb (id integer, payload text) PARTITION BY HASH (id);
+CREATE TABLE prtb1 PARTITION OF prtb FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE prtb2 PARTITION OF prtb FOR VALUES WITH (modulus 2, remainder 1);
+CREATE UNIQUE INDEX ON prtb1 (id);
+CREATE UNIQUE INDEX ON prtb2 (id);
+INSERT INTO prtb (id, payload)
+ (SELECT *, ('def' || id)::text AS payload
+ FROM generate_series(1,1000) AS id);
+
+CREATE TABLE e (id integer, payload text) PARTITION BY HASH (id);
+CREATE TABLE e1 PARTITION OF e FOR VALUES WITH (modulus 2, remainder 0);
+CREATE TABLE e2 PARTITION OF e FOR VALUES WITH (modulus 2, remainder 1);
+INSERT INTO e (id, payload)
+ (SELECT *, ('ghi' || id)::text AS payload
+ FROM generate_series(1,1000) AS id);
+CREATE UNIQUE INDEX ON e1 (id);
+CREATE UNIQUE INDEX ON e2 (id);
+
+ANALYZE prta,prtb,e;
+
+EXPLAIN (COSTS OFF)
+SELECT * FROM prta,prtb WHERE prta.id=prtb.id;
+
+EXPLAIN (COSTS OFF)
+SELECT * FROM prta,prtb,e WHERE prta.id=prtb.id AND prta.id=e.id;
+
-- semi join
EXPLAIN (COSTS OFF)
SELECT t1.* FROM prt1_adv t1 WHERE EXISTS (SELECT 1 FROM prt2_adv t2 WHERE t1.a = t2.b) AND t1.b = 0 ORDER BY t1.a;
--
2.31.1
^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: Asymmetric partition-wise JOIN
2019-08-12 06:03 Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-08-22 16:05 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-08-23 22:01 ` Re: Asymmetric partition-wise JOIN Thomas Munro <[email protected]>
2019-08-24 08:33 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-12-01 03:24 ` Re: Asymmetric partition-wise JOIN Michael Paquier <[email protected]>
2019-12-27 07:34 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2020-07-01 09:10 ` Re: Asymmetric partition-wise JOIN Daniel Gustafsson <[email protected]>
2020-08-21 06:02 ` Re: Asymmetric partition-wise JOIN Andrey V. Lepikhov <[email protected]>
2020-11-09 10:53 ` Re: Asymmetric partition-wise JOIN Anastasia Lubennikova <[email protected]>
2020-11-30 14:43 ` Re: Asymmetric partition-wise JOIN Anastasia Lubennikova <[email protected]>
2021-04-30 03:10 ` Re: Asymmetric partition-wise JOIN Andrey V. Lepikhov <[email protected]>
2021-05-27 04:27 ` Re: Asymmetric partition-wise JOIN Andrey Lepikhov <[email protected]>
2021-06-18 12:02 ` Re: Asymmetric partition-wise JOIN Alexander Pyhalov <[email protected]>
2021-07-05 09:57 ` Re: Asymmetric partition-wise JOIN Andrey Lepikhov <[email protected]>
2021-07-05 20:15 ` Re: Asymmetric partition-wise JOIN Zhihong Yu <[email protected]>
2021-07-15 06:32 ` Re: Asymmetric partition-wise JOIN Andrey Lepikhov <[email protected]>
@ 2021-07-15 13:16 ` Ibrar Ahmed <[email protected]>
2021-09-09 09:50 ` Re: Asymmetric partition-wise JOIN Aleksander Alekseev <[email protected]>
0 siblings, 1 reply; 28+ messages in thread
From: Ibrar Ahmed @ 2021-07-15 13:16 UTC (permalink / raw)
To: Andrey Lepikhov <[email protected]>; Kohei KaiGai <[email protected]>; +Cc: Zhihong Yu <[email protected]>; Alexander Pyhalov <[email protected]>; Anastasia Lubennikova <[email protected]>; Daniel Gustafsson <[email protected]>; Michael Paquier <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers
On Thu, Jul 15, 2021 at 11:32 AM Andrey Lepikhov <[email protected]>
wrote:
> On 5/7/21 23:15, Zhihong Yu wrote:
> > On Mon, Jul 5, 2021 at 2:57 AM Andrey Lepikhov
> > <[email protected] <mailto:[email protected]>> wrote:
> > + * Can't imagine situation when join relation already
> > exists. But in
> > + * the 'partition_join' regression test it happens.
> > + * It may be an indicator of possible problems.
> >
> > Should a log be added in the above case ?
> I worked more on this case and found more serious mistake. During
> population of additional paths on the existed RelOptInfo we can remove
> some previously generated paths that pointed from a higher-level list of
> subplans and it could cause to lost of subplan links. I prohibit such
> situation (you can read comments in the new version of the patch).
> Also, choosing of a cheapest path after appendrel creation was added.
> Unstable tests were fixed.
>
> --
> regards,
> Andrey Lepikhov
> Postgres Professional
>
Patch is failing the regression, can you please take a look at that.
partition_join ... FAILED 6328 ms
--
Ibrar Ahmed
^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: Asymmetric partition-wise JOIN
2019-08-12 06:03 Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-08-22 16:05 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-08-23 22:01 ` Re: Asymmetric partition-wise JOIN Thomas Munro <[email protected]>
2019-08-24 08:33 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-12-01 03:24 ` Re: Asymmetric partition-wise JOIN Michael Paquier <[email protected]>
2019-12-27 07:34 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2020-07-01 09:10 ` Re: Asymmetric partition-wise JOIN Daniel Gustafsson <[email protected]>
2020-08-21 06:02 ` Re: Asymmetric partition-wise JOIN Andrey V. Lepikhov <[email protected]>
2020-11-09 10:53 ` Re: Asymmetric partition-wise JOIN Anastasia Lubennikova <[email protected]>
2020-11-30 14:43 ` Re: Asymmetric partition-wise JOIN Anastasia Lubennikova <[email protected]>
2021-04-30 03:10 ` Re: Asymmetric partition-wise JOIN Andrey V. Lepikhov <[email protected]>
2021-05-27 04:27 ` Re: Asymmetric partition-wise JOIN Andrey Lepikhov <[email protected]>
2021-06-18 12:02 ` Re: Asymmetric partition-wise JOIN Alexander Pyhalov <[email protected]>
2021-07-05 09:57 ` Re: Asymmetric partition-wise JOIN Andrey Lepikhov <[email protected]>
2021-07-05 20:15 ` Re: Asymmetric partition-wise JOIN Zhihong Yu <[email protected]>
2021-07-15 06:32 ` Re: Asymmetric partition-wise JOIN Andrey Lepikhov <[email protected]>
2021-07-15 13:16 ` Re: Asymmetric partition-wise JOIN Ibrar Ahmed <[email protected]>
@ 2021-09-09 09:50 ` Aleksander Alekseev <[email protected]>
2021-09-09 15:38 ` Re: Asymmetric partition-wise JOIN Jaime Casanova <[email protected]>
0 siblings, 1 reply; 28+ messages in thread
From: Aleksander Alekseev @ 2021-09-09 09:50 UTC (permalink / raw)
To: [email protected]; +Cc: KaiGai Kohei <[email protected]>; Andrey Lepikhov <[email protected]>
It looks like this patch needs to be updated. According to http://cfbot.cputube.org/ it applies but doesn't pass any tests. Changing the status to save time for reviewers.
The new status of this patch is: Waiting on Author
^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: Asymmetric partition-wise JOIN
2019-08-12 06:03 Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-08-22 16:05 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-08-23 22:01 ` Re: Asymmetric partition-wise JOIN Thomas Munro <[email protected]>
2019-08-24 08:33 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-12-01 03:24 ` Re: Asymmetric partition-wise JOIN Michael Paquier <[email protected]>
2019-12-27 07:34 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2020-07-01 09:10 ` Re: Asymmetric partition-wise JOIN Daniel Gustafsson <[email protected]>
2020-08-21 06:02 ` Re: Asymmetric partition-wise JOIN Andrey V. Lepikhov <[email protected]>
2020-11-09 10:53 ` Re: Asymmetric partition-wise JOIN Anastasia Lubennikova <[email protected]>
2020-11-30 14:43 ` Re: Asymmetric partition-wise JOIN Anastasia Lubennikova <[email protected]>
2021-04-30 03:10 ` Re: Asymmetric partition-wise JOIN Andrey V. Lepikhov <[email protected]>
2021-05-27 04:27 ` Re: Asymmetric partition-wise JOIN Andrey Lepikhov <[email protected]>
2021-06-18 12:02 ` Re: Asymmetric partition-wise JOIN Alexander Pyhalov <[email protected]>
2021-07-05 09:57 ` Re: Asymmetric partition-wise JOIN Andrey Lepikhov <[email protected]>
2021-07-05 20:15 ` Re: Asymmetric partition-wise JOIN Zhihong Yu <[email protected]>
2021-07-15 06:32 ` Re: Asymmetric partition-wise JOIN Andrey Lepikhov <[email protected]>
2021-07-15 13:16 ` Re: Asymmetric partition-wise JOIN Ibrar Ahmed <[email protected]>
2021-09-09 09:50 ` Re: Asymmetric partition-wise JOIN Aleksander Alekseev <[email protected]>
@ 2021-09-09 15:38 ` Jaime Casanova <[email protected]>
2021-09-14 06:37 ` Re: Asymmetric partition-wise JOIN Andrey V. Lepikhov <[email protected]>
0 siblings, 1 reply; 28+ messages in thread
From: Jaime Casanova @ 2021-09-09 15:38 UTC (permalink / raw)
To: Aleksander Alekseev <[email protected]>; +Cc: [email protected]; KaiGai Kohei <[email protected]>; Andrey Lepikhov <[email protected]>
On Thu, Sep 09, 2021 at 09:50:46AM +0000, Aleksander Alekseev wrote:
> It looks like this patch needs to be updated. According to http://cfbot.cputube.org/ it applies but doesn't pass any tests. Changing the status to save time for reviewers.
>
> The new status of this patch is: Waiting on Author
Just to give some more info to work on I found this patch made postgres
crash with a segmentation fault.
"""
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x0000556e37ef1b55 in bms_equal (a=0x7f6e37a9c5b0, b=0x7f6e37a9c5b0) at bitmapset.c:126
126 if (shorter->words[i] != longer->words[i])
"""
attached are the query that triggers the crash and the backtrace.
--
Jaime Casanova
Director de Servicios Profesionales
SystemGuards - Consultores de PostgreSQL
#0 0x0000556e37ef1b55 in bms_equal (a=0x7f6e37a9c5b0, b=0x7f6e37a9c5b0) at bitmapset.c:126
shorter = 0x7f6e37a9c5b0
longer = 0x7f6e37a9c5b0
shortlen = 2139062143
longlen = 32622
i = 138057
#1 0x0000556e37fc9f5d in find_param_path_info (rel=0x556e3944c490, required_outer=0x7f6e37a9c5b0) at relnode.c:1634
ppi = 0x7f6e37a9cc08
lc__state = {l = 0x7f6e37a9cc40, i = 0}
lc = 0x7f6e37a5f898
#2 0x0000556e37fbeaeb in reparameterize_path_by_child (root=0x556e394340e0, path=0x7f6e37a9c6c8, child_rel=0x7f6e37a77ac0) at pathnode.c:4205
new_path = 0x7f6e37a89408
new_ppi = 0x7f6e37a77d30
old_ppi = 0x7f6e37a9cc08
required_outer = 0x7f6e37a9c5b0
#3 0x0000556e37f66545 in try_nestloop_path (root=0x556e394340e0, joinrel=0x7f6e37a88a80, outer_path=0x7f6e37a788e8, inner_path=0x7f6e37a9c6c8, pathkeys=0x0, jointype=JOIN_LEFT, extra=0x7ffde36f3e50)
at joinpath.c:662
required_outer = 0x0
workspace = {startup_cost = 46733.467499999999, total_cost = 1195882.0625, run_cost = 1149148.595, inner_run_cost = 0.020375816993464052, inner_rescan_run_cost = 0.020375816993464052,
outer_rows = 6.9224208015025731e-310, inner_rows = 6.9224208087218603e-310, outer_skip_rows = 6.9224208013235237e-310, inner_skip_rows = 4.640852265067508e-310, numbuckets = 960708832,
numbatches = 21870, inner_rows_total = 6.9224208047606396e-310}
innerrel = 0x556e3944c490
outerrel = 0x7f6e37a77ac0
innerrelids = 0x556e3944c2c8
outerrelids = 0x7f6e37a77d30
inner_paramrels = 0x7f6e37a9c5b0
outer_paramrels = 0x0
#4 0x0000556e37f67bf9 in match_unsorted_outer (root=0x556e394340e0, joinrel=0x7f6e37a88a80, outerrel=0x7f6e37a77ac0, innerrel=0x556e3944c490, jointype=JOIN_LEFT, extra=0x7ffde36f3e50) at joinpath.c:1702
innerpath = 0x7f6e37a9c6c8
mpath = 0x0
lc2__state = {l = 0x7f6e37a9d0a0, i = 1}
lc2 = 0x7f6e37a9d0c0
outerpath = 0x7f6e37a788e8
merge_pathkeys = 0x0
lc1__state = {l = 0x7f6e37a786c8, i = 0}
save_jointype = JOIN_LEFT
nestjoinOK = true
useallclauses = false
inner_cheapest_total = 0x7f6e37a9c3b0
matpath = 0x7f6e37a89370
lc1 = 0x7f6e37a786e0
__func__ = "match_unsorted_outer"
#5 0x0000556e37f65c14 in add_paths_to_joinrel (root=0x556e394340e0, joinrel=0x7f6e37a88a80, outerrel=0x7f6e37a77ac0, innerrel=0x556e3944c490, jointype=JOIN_LEFT, sjinfo=0x7f6e37a88630,
restrictlist=0x7f6e37a88a28) at joinpath.c:291
extra = {restrictlist = 0x7f6e37a88a28, mergeclause_list = 0x7f6e37a88e28, inner_unique = true, sjinfo = 0x7f6e37a88630, semifactors = {outer_match_frac = 0.00039215686274509802, match_count = 2550},
param_source_rels = 0x0}
mergejoin_allowed = true
lc = 0x0
joinrelids = 0x7f6e37a886e8
#6 0x0000556e37f6a3d9 in populate_joinrel_with_paths (root=0x556e394340e0, rel1=0x7f6e37a77ac0, rel2=0x556e3944c490, joinrel=0x7f6e37a88a80, sjinfo=0x7f6e37a88630, restrictlist=0x7f6e37a88a28)
at joinrels.c:825
__func__ = "populate_joinrel_with_paths"
#7 0x0000556e37f6bca5 in extract_asymmetric_partitionwise_subjoin (root=0x556e394340e0, joinrel=0x7f6e37a86bd8, append_path=0x7f6e37a7b6c8, inner_rel=0x556e3944c490, jointype=JOIN_LEFT, extra=0x7ffde36f4150)
at joinrels.c:1617
child_path = 0x7f6e37a788e8
child_joinrelids = 0x7f6e37a885e0
child_sjinfo = 0x7f6e37a88630
child_rel = 0x7f6e37a77ac0
parent_relids = 0x7f6e37a88608
child_joinrel = 0x7f6e37a88a80
child_restrictlist = 0x7f6e37a88a28
lc__state = {l = 0x7f6e37a7b8a8, i = 0}
result = 0x0
lc = 0x7f6e37a7b8c0
#8 0x0000556e37f6bfb8 in try_asymmetric_partitionwise_join (root=0x556e394340e0, joinrel=0x7f6e37a86bd8, outer_rel=0x7f6e37a65f68, inner_rel=0x556e3944c490, jointype=JOIN_LEFT, extra=0x7ffde36f4150)
at joinrels.c:1713
_save_exception_stack = 0x7ffde36f4b80
_save_context_stack = 0x0
_local_sigjmp_buf = {{__jmpbuf = {0, -1508331243234128262, 93931869773824, 140728419176608, 0, 0, -1508331245744419206, -4740322787736023430}, __mask_was_saved = 0, __saved_mask = {__val = {0,
140728419172688, 4294967296, 93931895571600, 140111356780392, 140111356914648, 93931895472352, 140111357005984, 4294967298, 140111356901176, 140110423130113, 140111356915312, 93930934763521,
140111356850992, 0, 140111357003464}}}}
_do_rethrow = false
join_rel_level_saved = 0x7f6e37aa0128
live_childrels = 0x0
append_path = 0x7f6e37a7b6c8
lc__state = {l = 0x7f6e37a663c0, i = 2}
lc = 0x7f6e37a663e8
#9 0x0000556e37f65d10 in add_paths_to_joinrel (root=0x556e394340e0, joinrel=0x7f6e37a86bd8, outerrel=0x7f6e37a65f68, innerrel=0x556e3944c490, jointype=JOIN_LEFT, sjinfo=0x556e3944f9a8,
restrictlist=0x7f6e37a86e70) at joinpath.c:344
extra = {restrictlist = 0x7f6e37a86e70, mergeclause_list = 0x7f6e37a86f98, inner_unique = true, sjinfo = 0x556e3944f9a8, semifactors = {outer_match_frac = 0.00039215686274509802, match_count = 2550},
param_source_rels = 0x0}
mergejoin_allowed = true
lc = 0x0
joinrelids = 0x7f6e37a86df0
#10 0x0000556e37f6a3d9 in populate_joinrel_with_paths (root=0x556e394340e0, rel1=0x7f6e37a65f68, rel2=0x556e3944c490, joinrel=0x7f6e37a86bd8, sjinfo=0x556e3944f9a8, restrictlist=0x7f6e37a86e70)
at joinrels.c:825
__func__ = "populate_joinrel_with_paths"
#11 0x0000556e37f6a213 in make_join_rel (root=0x556e394340e0, rel1=0x7f6e37a65f68, rel2=0x556e3944c490) at joinrels.c:761
joinrelids = 0x7f6e37a9cb88
sjinfo = 0x556e3944f9a8
reversed = false
sjinfo_data = {type = T_SpecialJoinInfo, min_lefthand = 0x556e3944c490, min_righthand = 0x7f6e37a65f68, syn_lefthand = 0x556e394340e0, syn_righthand = 0x7f6e37a662a8, jointype = JOIN_INNER,
lhs_strict = false, delay_upper_joins = false, semi_can_btree = false, semi_can_hash = false, semi_operators = 0x0, semi_rhs_exprs = 0x556e3944bb00}
joinrel = 0x7f6e37a86bd8
restrictlist = 0x7f6e37a86e70
#12 0x0000556e37f69708 in make_rels_by_clause_joins (root=0x556e394340e0, old_rel=0x7f6e37a65f68, other_rels_list=0x7f6e37aa00d0, other_rels=0x7f6e37aa00e8) at joinrels.c:313
other_rel = 0x556e3944c490
l__state = {l = 0x7f6e37aa00d0, i = 4}
l = 0x7f6e37aa0108
#13 0x0000556e37f691da in join_search_one_level (root=0x556e394340e0, level=4) at joinrels.c:124
other_rels_list = 0x7f6e37aa00d0
other_rels = 0x7f6e37aa00e8
old_rel = 0x7f6e37a65f68
r__state = {l = 0x7f6e37a63ec0, i = 1}
joinrels = 0x7f6e37aa0128
r = 0x7f6e37a7bd60
k = 32622
__func__ = "join_search_one_level"
#14 0x0000556e37f4bf87 in standard_join_search (root=0x556e394340e0, levels_needed=5, initial_rels=0x7f6e37aa00d0) at allpaths.c:3020
lc = 0x0
lev = 4
rel = 0x7f6e37a80030
__func__ = "standard_join_search"
#15 0x0000556e37f4bef7 in make_rel_from_joinlist (root=0x556e394340e0, joinlist=0x556e39452568) at allpaths.c:2951
levels_needed = 5
initial_rels = 0x7f6e37aa00d0
jl = 0x0
__func__ = "make_rel_from_joinlist"
#16 0x0000556e37f47a62 in make_one_rel (root=0x556e394340e0, joinlist=0x556e39452568) at allpaths.c:228
rel = 0x394340e0
rti = 21
total_pages = 146
#17 0x0000556e37f84d4e in query_planner (root=0x556e394340e0, qp_callback=0x556e37f8aa09 <standard_qp_callback>, qp_extra=0x7ffde36f4580) at planmain.c:276
parse = 0x556e3939a390
joinlist = 0x556e39452568
final_rel = 0x556e39449148
__func__ = "query_planner"
#18 0x0000556e37f87507 in grouping_planner (root=0x556e394340e0, tuple_fraction=0) at planner.c:1447
sort_input_targets = 0x0
sort_input_target_parallel_safe = false
grouping_target = 0x556e39448d50
scanjoin_target = 0x0
activeWindows = 0x0
qp_extra = {activeWindows = 0x0, groupClause = 0x0}
sort_input_targets_contain_srfs = 0x556e3944a240
have_grouping = false
wflists = 0x0
gset_data = 0x0
sort_input_target = 0x556e39449d88
grouping_targets = 0x0
grouping_target_parallel_safe = false
scanjoin_targets = 0x556e3944a0b8
scanjoin_target_parallel_safe = false
grouping_targets_contain_srfs = 0x556e39449df8
scanjoin_targets_contain_srfs = 0x0
scanjoin_target_same_exprs = false
parse = 0x556e3939a390
offset_est = 0
count_est = 0
limit_tuples = -1
have_postponed_srfs = false
final_target = 0x7ffde36f4710
final_targets = 0x556e394340e0
final_targets_contain_srfs = 0x0
final_target_parallel_safe = false
current_rel = 0x7ffde36f4710
final_rel = 0x7ffde36f4650
extra = {limit_needed = 224, limit_tuples = 4.6408511946627746e-310, count_est = 0, offset_est = 0}
lc = 0x0
__func__ = "grouping_planner"
#19 0x0000556e37f86bae in subquery_planner (glob=0x556e39448e38, parse=0x556e3939a390, parent_root=0x0, hasRecursion=false, tuple_fraction=0) at planner.c:1025
root = 0x556e394340e0
newWithCheckOptions = 0x0
newHaving = 0x0
hasOuterJoins = true
hasResultRTEs = false
final_rel = 0x910100004000
l = 0x0
#20 0x0000556e37f8539b in standard_planner (parse=0x556e3939a390,
query_string=0x556e393728f0 "select \n 1 \nfrom fkpart5.fk as ref_1\n inner join fkpart5.pk31 as sample_0 tablesample system (4.1) on (ref_1.a = sample_0.a )\n inner join public.pagg_tab_para_p2 as ref_3 on (ref_1.a > 0.3)\n"..., cursorOptions=2048, boundParams=0x0) at planner.c:406
result = 0x556e382990f3 <palloc+273>
glob = 0x556e39448e38
tuple_fraction = 0
root = 0x556e37f122ed <lappend+141>
final_rel = 0x7ffde36f4930
best_path = 0x556e39448de0
top_plan = 0x7ffde36f49c0
lp = 0x556e39437800
lr = 0x556e37f11e04 <new_list+103>
#21 0x0000556e37f85150 in planner (parse=0x556e3939a390,
query_string=0x556e393728f0 "select \n 1 \nfrom fkpart5.fk as ref_1\n inner join fkpart5.pk31 as sample_0 tablesample system (4.1) on (ref_1.a = sample_0.a )\n inner join public.pagg_tab_para_p2 as ref_3 on (ref_1.a > 0.3)\n"..., cursorOptions=2048, boundParams=0x0) at planner.c:277
result = 0x556e3939a390
#22 0x0000556e380bf572 in pg_plan_query (querytree=0x556e3939a390,
query_string=0x556e393728f0 "select \n 1 \nfrom fkpart5.fk as ref_1\n inner join fkpart5.pk31 as sample_0 tablesample system (4.1) on (ref_1.a = sample_0.a )\n inner join public.pagg_tab_para_p2 as ref_3 on (ref_1.a > 0.3)\n"..., cursorOptions=2048, boundParams=0x0) at postgres.c:847
plan = 0xeb380bf4a3
#23 0x0000556e380bf6bc in pg_plan_queries (querytrees=0x556e39448de0,
query_string=0x556e393728f0 "select \n 1 \nfrom fkpart5.fk as ref_1\n inner join fkpart5.pk31 as sample_0 tablesample system (4.1) on (ref_1.a = sample_0.a )\n inner join public.pagg_tab_para_p2 as ref_3 on (ref_1.a > 0.3)\n"..., cursorOptions=2048, boundParams=0x0) at postgres.c:939
query = 0x556e3939a390
stmt = 0x556e3939a1e8
query_list__state = {l = 0x556e39448de0, i = 0}
stmt_list = 0x0
query_list = 0x556e39448df8
#24 0x0000556e380bfa2e in exec_simple_query (
query_string=0x556e393728f0 "select \n 1 \nfrom fkpart5.fk as ref_1\n inner join fkpart5.pk31 as sample_0 tablesample system (4.1) on (ref_1.a = sample_0.a )\n inner join public.pagg_tab_para_p2 as ref_3 on (ref_1.a > 0.3)\n"...) at postgres.c:1133
snapshot_set = true
per_parsetree_context = 0x0
plantree_list = 0x556e37bb2000 <_start>
parsetree = 0x556e3939a1e8
commandTag = CMDTAG_SELECT
qc = {commandTag = 3815721648, nprocessed = 93931876865050}
querytree_list = 0x556e39448de0
portal = 0x0
receiver = 0x556e38274314 <pg_any_to_server+88>
format = 0
parsetree_item__state = {l = 0x556e3939a220, i = 0}
dest = DestRemote
oldcontext = 0x556e393f9710
parsetree_list = 0x556e3939a220
parsetree_item = 0x556e3939a238
save_log_statement_stats = false
was_logged = false
use_implicit_block = false
msec_str = "\360(79nU\000\000F\001\000\000\006\000\000\000\300Jo\343\375\177\000\000\061\343\356\067F\001\000"
__func__ = "exec_simple_query"
#25 0x0000556e380c4380 in PostgresMain (argc=1, argv=0x7ffde36f4d00, dbname=0x556e3939f580 "regression", username=0x556e3939f558 "jcasanov") at postgres.c:4488
query_string = 0x556e393728f0 "select \n 1 \nfrom fkpart5.fk as ref_1\n inner join fkpart5.pk31 as sample_0 tablesample system (4.1) on (ref_1.a = sample_0.a )\n inner join public.pagg_tab_para_p2 as ref_3 on (ref_1.a > 0.3)\n"...
firstchar = 81
input_message = {
data = 0x556e393728f0 "select \n 1 \nfrom fkpart5.fk as ref_1\n inner join fkpart5.pk31 as sample_0 tablesample system (4.1) on (ref_1.a = sample_0.a )\n inner join public.pagg_tab_para_p2 as ref_3 on (ref_1.a > 0.3)\n"..., len = 327, maxlen = 1024, cursor = 327}
local_sigjmp_buf = {{__jmpbuf = {0, -1508331243102007686, 93931869773824, 140728419176608, 0, 0, -1508331242955207046, -4740352521772069254}, __mask_was_saved = 1, __saved_mask = {__val = {4194304, 0,
0, 0, 0, 0, 0, 0, 0, 0, 1099511627520, 0, 0, 4294901760, 18446462598732840960, 0}}}}
send_ready_for_query = false
idle_in_transaction_timeout_enabled = false
idle_session_timeout_enabled = false
__func__ = "PostgresMain"
#26 0x0000556e37ff9e84 in BackendRun (port=0x556e39398e70) at postmaster.c:4521
av = {0x556e3840927f "postgres", 0x0}
ac = 1
#27 0x0000556e37ff97b3 in BackendStartup (port=0x556e39398e70) at postmaster.c:4243
bn = 0x556e39398e40
pid = 0
__func__ = "BackendStartup"
#28 0x0000556e37ff5b04 in ServerLoop () at postmaster.c:1765
port = 0x556e39398e70
i = 2
rmask = {fds_bits = {128, 0 <repeats 15 times>}}
selres = 1
now = 1631170136
readmask = {fds_bits = {224, 0 <repeats 15 times>}}
nSockets = 8
last_lockfile_recheck_time = 1631170106
last_touch_time = 1631169617
__func__ = "ServerLoop"
#29 0x0000556e37ff5353 in PostmasterMain (argc=3, argv=0x556e3936bef0) at postmaster.c:1437
opt = -1
status = 0
userDoption = 0x556e3938f810 "data"
listen_addr_saved = true
i = 64
output_config_variable = 0x0
__func__ = "PostmasterMain"
#30 0x0000556e37ef159f in main (argc=3, argv=0x556e3936bef0) at main.c:199
do_check_root = true
Attachments:
[application/x-sql] bug.sql (365B, ../../20210909153833.GA6514@ahch-to/2-bug.sql)
download
[text/plain] gdb.txt (14.7K, ../../20210909153833.GA6514@ahch-to/3-gdb.txt)
download | inline:
#0 0x0000556e37ef1b55 in bms_equal (a=0x7f6e37a9c5b0, b=0x7f6e37a9c5b0) at bitmapset.c:126
shorter = 0x7f6e37a9c5b0
longer = 0x7f6e37a9c5b0
shortlen = 2139062143
longlen = 32622
i = 138057
#1 0x0000556e37fc9f5d in find_param_path_info (rel=0x556e3944c490, required_outer=0x7f6e37a9c5b0) at relnode.c:1634
ppi = 0x7f6e37a9cc08
lc__state = {l = 0x7f6e37a9cc40, i = 0}
lc = 0x7f6e37a5f898
#2 0x0000556e37fbeaeb in reparameterize_path_by_child (root=0x556e394340e0, path=0x7f6e37a9c6c8, child_rel=0x7f6e37a77ac0) at pathnode.c:4205
new_path = 0x7f6e37a89408
new_ppi = 0x7f6e37a77d30
old_ppi = 0x7f6e37a9cc08
required_outer = 0x7f6e37a9c5b0
#3 0x0000556e37f66545 in try_nestloop_path (root=0x556e394340e0, joinrel=0x7f6e37a88a80, outer_path=0x7f6e37a788e8, inner_path=0x7f6e37a9c6c8, pathkeys=0x0, jointype=JOIN_LEFT, extra=0x7ffde36f3e50)
at joinpath.c:662
required_outer = 0x0
workspace = {startup_cost = 46733.467499999999, total_cost = 1195882.0625, run_cost = 1149148.595, inner_run_cost = 0.020375816993464052, inner_rescan_run_cost = 0.020375816993464052,
outer_rows = 6.9224208015025731e-310, inner_rows = 6.9224208087218603e-310, outer_skip_rows = 6.9224208013235237e-310, inner_skip_rows = 4.640852265067508e-310, numbuckets = 960708832,
numbatches = 21870, inner_rows_total = 6.9224208047606396e-310}
innerrel = 0x556e3944c490
outerrel = 0x7f6e37a77ac0
innerrelids = 0x556e3944c2c8
outerrelids = 0x7f6e37a77d30
inner_paramrels = 0x7f6e37a9c5b0
outer_paramrels = 0x0
#4 0x0000556e37f67bf9 in match_unsorted_outer (root=0x556e394340e0, joinrel=0x7f6e37a88a80, outerrel=0x7f6e37a77ac0, innerrel=0x556e3944c490, jointype=JOIN_LEFT, extra=0x7ffde36f3e50) at joinpath.c:1702
innerpath = 0x7f6e37a9c6c8
mpath = 0x0
lc2__state = {l = 0x7f6e37a9d0a0, i = 1}
lc2 = 0x7f6e37a9d0c0
outerpath = 0x7f6e37a788e8
merge_pathkeys = 0x0
lc1__state = {l = 0x7f6e37a786c8, i = 0}
save_jointype = JOIN_LEFT
nestjoinOK = true
useallclauses = false
inner_cheapest_total = 0x7f6e37a9c3b0
matpath = 0x7f6e37a89370
lc1 = 0x7f6e37a786e0
__func__ = "match_unsorted_outer"
#5 0x0000556e37f65c14 in add_paths_to_joinrel (root=0x556e394340e0, joinrel=0x7f6e37a88a80, outerrel=0x7f6e37a77ac0, innerrel=0x556e3944c490, jointype=JOIN_LEFT, sjinfo=0x7f6e37a88630,
restrictlist=0x7f6e37a88a28) at joinpath.c:291
extra = {restrictlist = 0x7f6e37a88a28, mergeclause_list = 0x7f6e37a88e28, inner_unique = true, sjinfo = 0x7f6e37a88630, semifactors = {outer_match_frac = 0.00039215686274509802, match_count = 2550},
param_source_rels = 0x0}
mergejoin_allowed = true
lc = 0x0
joinrelids = 0x7f6e37a886e8
#6 0x0000556e37f6a3d9 in populate_joinrel_with_paths (root=0x556e394340e0, rel1=0x7f6e37a77ac0, rel2=0x556e3944c490, joinrel=0x7f6e37a88a80, sjinfo=0x7f6e37a88630, restrictlist=0x7f6e37a88a28)
at joinrels.c:825
__func__ = "populate_joinrel_with_paths"
#7 0x0000556e37f6bca5 in extract_asymmetric_partitionwise_subjoin (root=0x556e394340e0, joinrel=0x7f6e37a86bd8, append_path=0x7f6e37a7b6c8, inner_rel=0x556e3944c490, jointype=JOIN_LEFT, extra=0x7ffde36f4150)
at joinrels.c:1617
child_path = 0x7f6e37a788e8
child_joinrelids = 0x7f6e37a885e0
child_sjinfo = 0x7f6e37a88630
child_rel = 0x7f6e37a77ac0
parent_relids = 0x7f6e37a88608
child_joinrel = 0x7f6e37a88a80
child_restrictlist = 0x7f6e37a88a28
lc__state = {l = 0x7f6e37a7b8a8, i = 0}
result = 0x0
lc = 0x7f6e37a7b8c0
#8 0x0000556e37f6bfb8 in try_asymmetric_partitionwise_join (root=0x556e394340e0, joinrel=0x7f6e37a86bd8, outer_rel=0x7f6e37a65f68, inner_rel=0x556e3944c490, jointype=JOIN_LEFT, extra=0x7ffde36f4150)
at joinrels.c:1713
_save_exception_stack = 0x7ffde36f4b80
_save_context_stack = 0x0
_local_sigjmp_buf = {{__jmpbuf = {0, -1508331243234128262, 93931869773824, 140728419176608, 0, 0, -1508331245744419206, -4740322787736023430}, __mask_was_saved = 0, __saved_mask = {__val = {0,
140728419172688, 4294967296, 93931895571600, 140111356780392, 140111356914648, 93931895472352, 140111357005984, 4294967298, 140111356901176, 140110423130113, 140111356915312, 93930934763521,
140111356850992, 0, 140111357003464}}}}
_do_rethrow = false
join_rel_level_saved = 0x7f6e37aa0128
live_childrels = 0x0
append_path = 0x7f6e37a7b6c8
lc__state = {l = 0x7f6e37a663c0, i = 2}
lc = 0x7f6e37a663e8
#9 0x0000556e37f65d10 in add_paths_to_joinrel (root=0x556e394340e0, joinrel=0x7f6e37a86bd8, outerrel=0x7f6e37a65f68, innerrel=0x556e3944c490, jointype=JOIN_LEFT, sjinfo=0x556e3944f9a8,
restrictlist=0x7f6e37a86e70) at joinpath.c:344
extra = {restrictlist = 0x7f6e37a86e70, mergeclause_list = 0x7f6e37a86f98, inner_unique = true, sjinfo = 0x556e3944f9a8, semifactors = {outer_match_frac = 0.00039215686274509802, match_count = 2550},
param_source_rels = 0x0}
mergejoin_allowed = true
lc = 0x0
joinrelids = 0x7f6e37a86df0
#10 0x0000556e37f6a3d9 in populate_joinrel_with_paths (root=0x556e394340e0, rel1=0x7f6e37a65f68, rel2=0x556e3944c490, joinrel=0x7f6e37a86bd8, sjinfo=0x556e3944f9a8, restrictlist=0x7f6e37a86e70)
at joinrels.c:825
__func__ = "populate_joinrel_with_paths"
#11 0x0000556e37f6a213 in make_join_rel (root=0x556e394340e0, rel1=0x7f6e37a65f68, rel2=0x556e3944c490) at joinrels.c:761
joinrelids = 0x7f6e37a9cb88
sjinfo = 0x556e3944f9a8
reversed = false
sjinfo_data = {type = T_SpecialJoinInfo, min_lefthand = 0x556e3944c490, min_righthand = 0x7f6e37a65f68, syn_lefthand = 0x556e394340e0, syn_righthand = 0x7f6e37a662a8, jointype = JOIN_INNER,
lhs_strict = false, delay_upper_joins = false, semi_can_btree = false, semi_can_hash = false, semi_operators = 0x0, semi_rhs_exprs = 0x556e3944bb00}
joinrel = 0x7f6e37a86bd8
restrictlist = 0x7f6e37a86e70
#12 0x0000556e37f69708 in make_rels_by_clause_joins (root=0x556e394340e0, old_rel=0x7f6e37a65f68, other_rels_list=0x7f6e37aa00d0, other_rels=0x7f6e37aa00e8) at joinrels.c:313
other_rel = 0x556e3944c490
l__state = {l = 0x7f6e37aa00d0, i = 4}
l = 0x7f6e37aa0108
#13 0x0000556e37f691da in join_search_one_level (root=0x556e394340e0, level=4) at joinrels.c:124
other_rels_list = 0x7f6e37aa00d0
other_rels = 0x7f6e37aa00e8
old_rel = 0x7f6e37a65f68
r__state = {l = 0x7f6e37a63ec0, i = 1}
joinrels = 0x7f6e37aa0128
r = 0x7f6e37a7bd60
k = 32622
__func__ = "join_search_one_level"
#14 0x0000556e37f4bf87 in standard_join_search (root=0x556e394340e0, levels_needed=5, initial_rels=0x7f6e37aa00d0) at allpaths.c:3020
lc = 0x0
lev = 4
rel = 0x7f6e37a80030
__func__ = "standard_join_search"
#15 0x0000556e37f4bef7 in make_rel_from_joinlist (root=0x556e394340e0, joinlist=0x556e39452568) at allpaths.c:2951
levels_needed = 5
initial_rels = 0x7f6e37aa00d0
jl = 0x0
__func__ = "make_rel_from_joinlist"
#16 0x0000556e37f47a62 in make_one_rel (root=0x556e394340e0, joinlist=0x556e39452568) at allpaths.c:228
rel = 0x394340e0
rti = 21
total_pages = 146
#17 0x0000556e37f84d4e in query_planner (root=0x556e394340e0, qp_callback=0x556e37f8aa09 <standard_qp_callback>, qp_extra=0x7ffde36f4580) at planmain.c:276
parse = 0x556e3939a390
joinlist = 0x556e39452568
final_rel = 0x556e39449148
__func__ = "query_planner"
#18 0x0000556e37f87507 in grouping_planner (root=0x556e394340e0, tuple_fraction=0) at planner.c:1447
sort_input_targets = 0x0
sort_input_target_parallel_safe = false
grouping_target = 0x556e39448d50
scanjoin_target = 0x0
activeWindows = 0x0
qp_extra = {activeWindows = 0x0, groupClause = 0x0}
sort_input_targets_contain_srfs = 0x556e3944a240
have_grouping = false
wflists = 0x0
gset_data = 0x0
sort_input_target = 0x556e39449d88
grouping_targets = 0x0
grouping_target_parallel_safe = false
scanjoin_targets = 0x556e3944a0b8
scanjoin_target_parallel_safe = false
grouping_targets_contain_srfs = 0x556e39449df8
scanjoin_targets_contain_srfs = 0x0
scanjoin_target_same_exprs = false
parse = 0x556e3939a390
offset_est = 0
count_est = 0
limit_tuples = -1
have_postponed_srfs = false
final_target = 0x7ffde36f4710
final_targets = 0x556e394340e0
final_targets_contain_srfs = 0x0
final_target_parallel_safe = false
current_rel = 0x7ffde36f4710
final_rel = 0x7ffde36f4650
extra = {limit_needed = 224, limit_tuples = 4.6408511946627746e-310, count_est = 0, offset_est = 0}
lc = 0x0
__func__ = "grouping_planner"
#19 0x0000556e37f86bae in subquery_planner (glob=0x556e39448e38, parse=0x556e3939a390, parent_root=0x0, hasRecursion=false, tuple_fraction=0) at planner.c:1025
root = 0x556e394340e0
newWithCheckOptions = 0x0
newHaving = 0x0
hasOuterJoins = true
hasResultRTEs = false
final_rel = 0x910100004000
l = 0x0
#20 0x0000556e37f8539b in standard_planner (parse=0x556e3939a390,
query_string=0x556e393728f0 "select \n 1 \nfrom fkpart5.fk as ref_1\n inner join fkpart5.pk31 as sample_0 tablesample system (4.1) on (ref_1.a = sample_0.a )\n inner join public.pagg_tab_para_p2 as ref_3 on (ref_1.a > 0.3)\n"..., cursorOptions=2048, boundParams=0x0) at planner.c:406
result = 0x556e382990f3 <palloc+273>
glob = 0x556e39448e38
tuple_fraction = 0
root = 0x556e37f122ed <lappend+141>
final_rel = 0x7ffde36f4930
best_path = 0x556e39448de0
top_plan = 0x7ffde36f49c0
lp = 0x556e39437800
lr = 0x556e37f11e04 <new_list+103>
#21 0x0000556e37f85150 in planner (parse=0x556e3939a390,
query_string=0x556e393728f0 "select \n 1 \nfrom fkpart5.fk as ref_1\n inner join fkpart5.pk31 as sample_0 tablesample system (4.1) on (ref_1.a = sample_0.a )\n inner join public.pagg_tab_para_p2 as ref_3 on (ref_1.a > 0.3)\n"..., cursorOptions=2048, boundParams=0x0) at planner.c:277
result = 0x556e3939a390
#22 0x0000556e380bf572 in pg_plan_query (querytree=0x556e3939a390,
query_string=0x556e393728f0 "select \n 1 \nfrom fkpart5.fk as ref_1\n inner join fkpart5.pk31 as sample_0 tablesample system (4.1) on (ref_1.a = sample_0.a )\n inner join public.pagg_tab_para_p2 as ref_3 on (ref_1.a > 0.3)\n"..., cursorOptions=2048, boundParams=0x0) at postgres.c:847
plan = 0xeb380bf4a3
#23 0x0000556e380bf6bc in pg_plan_queries (querytrees=0x556e39448de0,
query_string=0x556e393728f0 "select \n 1 \nfrom fkpart5.fk as ref_1\n inner join fkpart5.pk31 as sample_0 tablesample system (4.1) on (ref_1.a = sample_0.a )\n inner join public.pagg_tab_para_p2 as ref_3 on (ref_1.a > 0.3)\n"..., cursorOptions=2048, boundParams=0x0) at postgres.c:939
query = 0x556e3939a390
stmt = 0x556e3939a1e8
query_list__state = {l = 0x556e39448de0, i = 0}
stmt_list = 0x0
query_list = 0x556e39448df8
#24 0x0000556e380bfa2e in exec_simple_query (
query_string=0x556e393728f0 "select \n 1 \nfrom fkpart5.fk as ref_1\n inner join fkpart5.pk31 as sample_0 tablesample system (4.1) on (ref_1.a = sample_0.a )\n inner join public.pagg_tab_para_p2 as ref_3 on (ref_1.a > 0.3)\n"...) at postgres.c:1133
snapshot_set = true
per_parsetree_context = 0x0
plantree_list = 0x556e37bb2000 <_start>
parsetree = 0x556e3939a1e8
commandTag = CMDTAG_SELECT
qc = {commandTag = 3815721648, nprocessed = 93931876865050}
querytree_list = 0x556e39448de0
portal = 0x0
receiver = 0x556e38274314 <pg_any_to_server+88>
format = 0
parsetree_item__state = {l = 0x556e3939a220, i = 0}
dest = DestRemote
oldcontext = 0x556e393f9710
parsetree_list = 0x556e3939a220
parsetree_item = 0x556e3939a238
save_log_statement_stats = false
was_logged = false
use_implicit_block = false
msec_str = "\360(79nU\000\000F\001\000\000\006\000\000\000\300Jo\343\375\177\000\000\061\343\356\067F\001\000"
__func__ = "exec_simple_query"
#25 0x0000556e380c4380 in PostgresMain (argc=1, argv=0x7ffde36f4d00, dbname=0x556e3939f580 "regression", username=0x556e3939f558 "jcasanov") at postgres.c:4488
query_string = 0x556e393728f0 "select \n 1 \nfrom fkpart5.fk as ref_1\n inner join fkpart5.pk31 as sample_0 tablesample system (4.1) on (ref_1.a = sample_0.a )\n inner join public.pagg_tab_para_p2 as ref_3 on (ref_1.a > 0.3)\n"...
firstchar = 81
input_message = {
data = 0x556e393728f0 "select \n 1 \nfrom fkpart5.fk as ref_1\n inner join fkpart5.pk31 as sample_0 tablesample system (4.1) on (ref_1.a = sample_0.a )\n inner join public.pagg_tab_para_p2 as ref_3 on (ref_1.a > 0.3)\n"..., len = 327, maxlen = 1024, cursor = 327}
local_sigjmp_buf = {{__jmpbuf = {0, -1508331243102007686, 93931869773824, 140728419176608, 0, 0, -1508331242955207046, -4740352521772069254}, __mask_was_saved = 1, __saved_mask = {__val = {4194304, 0,
0, 0, 0, 0, 0, 0, 0, 0, 1099511627520, 0, 0, 4294901760, 18446462598732840960, 0}}}}
send_ready_for_query = false
idle_in_transaction_timeout_enabled = false
idle_session_timeout_enabled = false
__func__ = "PostgresMain"
#26 0x0000556e37ff9e84 in BackendRun (port=0x556e39398e70) at postmaster.c:4521
av = {0x556e3840927f "postgres", 0x0}
ac = 1
#27 0x0000556e37ff97b3 in BackendStartup (port=0x556e39398e70) at postmaster.c:4243
bn = 0x556e39398e40
pid = 0
__func__ = "BackendStartup"
#28 0x0000556e37ff5b04 in ServerLoop () at postmaster.c:1765
port = 0x556e39398e70
i = 2
rmask = {fds_bits = {128, 0 <repeats 15 times>}}
selres = 1
now = 1631170136
readmask = {fds_bits = {224, 0 <repeats 15 times>}}
nSockets = 8
last_lockfile_recheck_time = 1631170106
last_touch_time = 1631169617
__func__ = "ServerLoop"
#29 0x0000556e37ff5353 in PostmasterMain (argc=3, argv=0x556e3936bef0) at postmaster.c:1437
opt = -1
status = 0
userDoption = 0x556e3938f810 "data"
listen_addr_saved = true
i = 64
output_config_variable = 0x0
__func__ = "PostmasterMain"
#30 0x0000556e37ef159f in main (argc=3, argv=0x556e3936bef0) at main.c:199
do_check_root = true
^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: Asymmetric partition-wise JOIN
2019-08-12 06:03 Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-08-22 16:05 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-08-23 22:01 ` Re: Asymmetric partition-wise JOIN Thomas Munro <[email protected]>
2019-08-24 08:33 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-12-01 03:24 ` Re: Asymmetric partition-wise JOIN Michael Paquier <[email protected]>
2019-12-27 07:34 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2020-07-01 09:10 ` Re: Asymmetric partition-wise JOIN Daniel Gustafsson <[email protected]>
2020-08-21 06:02 ` Re: Asymmetric partition-wise JOIN Andrey V. Lepikhov <[email protected]>
2020-11-09 10:53 ` Re: Asymmetric partition-wise JOIN Anastasia Lubennikova <[email protected]>
2020-11-30 14:43 ` Re: Asymmetric partition-wise JOIN Anastasia Lubennikova <[email protected]>
2021-04-30 03:10 ` Re: Asymmetric partition-wise JOIN Andrey V. Lepikhov <[email protected]>
2021-05-27 04:27 ` Re: Asymmetric partition-wise JOIN Andrey Lepikhov <[email protected]>
2021-06-18 12:02 ` Re: Asymmetric partition-wise JOIN Alexander Pyhalov <[email protected]>
2021-07-05 09:57 ` Re: Asymmetric partition-wise JOIN Andrey Lepikhov <[email protected]>
2021-07-05 20:15 ` Re: Asymmetric partition-wise JOIN Zhihong Yu <[email protected]>
2021-07-15 06:32 ` Re: Asymmetric partition-wise JOIN Andrey Lepikhov <[email protected]>
2021-07-15 13:16 ` Re: Asymmetric partition-wise JOIN Ibrar Ahmed <[email protected]>
2021-09-09 09:50 ` Re: Asymmetric partition-wise JOIN Aleksander Alekseev <[email protected]>
2021-09-09 15:38 ` Re: Asymmetric partition-wise JOIN Jaime Casanova <[email protected]>
@ 2021-09-14 06:37 ` Andrey V. Lepikhov <[email protected]>
2021-09-15 06:31 ` Re: Asymmetric partition-wise JOIN Andrey Lepikhov <[email protected]>
0 siblings, 1 reply; 28+ messages in thread
From: Andrey V. Lepikhov @ 2021-09-14 06:37 UTC (permalink / raw)
To: Jaime Casanova <[email protected]>; Aleksander Alekseev <[email protected]>; +Cc: [email protected]; KaiGai Kohei <[email protected]>
On 9/9/21 8:38 PM, Jaime Casanova wrote:
> On Thu, Sep 09, 2021 at 09:50:46AM +0000, Aleksander Alekseev wrote:
>> It looks like this patch needs to be updated. According to http://cfbot.cputube.org/ it applies but doesn't pass any tests. Changing the status to save time for reviewers.
>>
>> The new status of this patch is: Waiting on Author
>
> Just to give some more info to work on I found this patch made postgres
> crash with a segmentation fault.
>
> """
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0 0x0000556e37ef1b55 in bms_equal (a=0x7f6e37a9c5b0, b=0x7f6e37a9c5b0) at bitmapset.c:126
> 126 if (shorter->words[i] != longer->words[i])
> """
>
> attached are the query that triggers the crash and the backtrace.
>
Thank you for this good catch!
The problem was in the adjust_child_relids_multilevel routine. The
tmp_result variable sometimes points to original required_outer.
This patch adds new ways which optimizer can generate plans. One
possible way is optimizer reparameterizes an inner by a plain relation
from the outer (maybe as a result of join of the plain relation and
partitioned relation). In this case we have to compare tmp_result with
original pointer to realize, it was changed or not.
The patch in attachment fixes this problem. Additional regression test
added.
--
regards,
Andrey Lepikhov
Postgres Professional
^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: Asymmetric partition-wise JOIN
2019-08-12 06:03 Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-08-22 16:05 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-08-23 22:01 ` Re: Asymmetric partition-wise JOIN Thomas Munro <[email protected]>
2019-08-24 08:33 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-12-01 03:24 ` Re: Asymmetric partition-wise JOIN Michael Paquier <[email protected]>
2019-12-27 07:34 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2020-07-01 09:10 ` Re: Asymmetric partition-wise JOIN Daniel Gustafsson <[email protected]>
2020-08-21 06:02 ` Re: Asymmetric partition-wise JOIN Andrey V. Lepikhov <[email protected]>
2020-11-09 10:53 ` Re: Asymmetric partition-wise JOIN Anastasia Lubennikova <[email protected]>
2020-11-30 14:43 ` Re: Asymmetric partition-wise JOIN Anastasia Lubennikova <[email protected]>
2021-04-30 03:10 ` Re: Asymmetric partition-wise JOIN Andrey V. Lepikhov <[email protected]>
2021-05-27 04:27 ` Re: Asymmetric partition-wise JOIN Andrey Lepikhov <[email protected]>
2021-06-18 12:02 ` Re: Asymmetric partition-wise JOIN Alexander Pyhalov <[email protected]>
2021-07-05 09:57 ` Re: Asymmetric partition-wise JOIN Andrey Lepikhov <[email protected]>
2021-07-05 20:15 ` Re: Asymmetric partition-wise JOIN Zhihong Yu <[email protected]>
2021-07-15 06:32 ` Re: Asymmetric partition-wise JOIN Andrey Lepikhov <[email protected]>
2021-07-15 13:16 ` Re: Asymmetric partition-wise JOIN Ibrar Ahmed <[email protected]>
2021-09-09 09:50 ` Re: Asymmetric partition-wise JOIN Aleksander Alekseev <[email protected]>
2021-09-09 15:38 ` Re: Asymmetric partition-wise JOIN Jaime Casanova <[email protected]>
2021-09-14 06:37 ` Re: Asymmetric partition-wise JOIN Andrey V. Lepikhov <[email protected]>
@ 2021-09-15 06:31 ` Andrey Lepikhov <[email protected]>
0 siblings, 0 replies; 28+ messages in thread
From: Andrey Lepikhov @ 2021-09-15 06:31 UTC (permalink / raw)
To: Jaime Casanova <[email protected]>; Aleksander Alekseev <[email protected]>; +Cc: [email protected]; KaiGai Kohei <[email protected]>
On 14/9/21 11:37, Andrey V. Lepikhov wrote:
> Thank you for this good catch!
> The problem was in the adjust_child_relids_multilevel routine. The
> tmp_result variable sometimes points to original required_outer.
> This patch adds new ways which optimizer can generate plans. One
> possible way is optimizer reparameterizes an inner by a plain relation
> from the outer (maybe as a result of join of the plain relation and
> partitioned relation). In this case we have to compare tmp_result with
> original pointer to realize, it was changed or not.
> The patch in attachment fixes this problem. Additional regression test
> added.
>
I thought more and realized there isn't necessary to recurse in the
adjust_child_relids_multilevel() routine if required_outer contains only
normal_relids.
Also, regression tests were improved a bit.
--
regards,
Andrey Lepikhov
Postgres Professional
^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: Asymmetric partition-wise JOIN
2019-08-12 06:03 Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-08-22 16:05 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-08-23 22:01 ` Re: Asymmetric partition-wise JOIN Thomas Munro <[email protected]>
2019-08-24 08:33 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-12-01 03:24 ` Re: Asymmetric partition-wise JOIN Michael Paquier <[email protected]>
2019-12-27 07:34 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
@ 2020-07-06 08:46 ` Andrey V. Lepikhov <[email protected]>
2 siblings, 0 replies; 28+ messages in thread
From: Andrey V. Lepikhov @ 2020-07-06 08:46 UTC (permalink / raw)
To: Kohei KaiGai <[email protected]>; Michael Paquier <[email protected]>; +Cc: Thomas Munro <[email protected]>; pgsql-hackers
On 12/27/19 12:34 PM, Kohei KaiGai wrote:
> The attached v2 fixed the problem, and regression test finished correctly.
Using your patch I saw incorrect value of predicted rows at the top node
of the plan: "Append (cost=270.02..35165.37 rows=40004 width=16)"
Full explain of the query plan see in attachment -
explain_with_asymmetric.sql
if I disable enable_partitionwise_join then:
"Hash Join (cost=270.02..38855.25 rows=10001 width=16)"
Full explain - explain_no_asymmetric.sql
I thought that is the case of incorrect usage of cached values of
norm_selec, but it is a corner-case problem of the eqjoinsel() routine :
selectivity = 1/size_of_larger_relation; (selfuncs.c:2567)
tuples = selectivity * outer_tuples * inner_tuples; (costsize.c:4607)
i.e. number of tuples depends only on size of smaller relation.
It is not a bug of your patch but I think you need to know because it
may affect on planner decision.
===
P.S. Test case:
CREATE TABLE t0 (a serial, b int);
INSERT INTO t0 (b) (SELECT * FROM generate_series(1e4, 2e4) as g);
CREATE TABLE parts (a serial, b int) PARTITION BY HASH(a)
INSERT INTO parts (b) (SELECT * FROM generate_series(1, 1e6) as g);
--
regards,
Andrey Lepikhov
Postgres Professional
Attachments:
[application/sql] explain_with_asymmetric.sql (1.3K, ../../[email protected]/2-explain_with_asymmetric.sql)
download
[application/sql] explain_no_asymmetric.sql (612B, ../../[email protected]/3-explain_no_asymmetric.sql)
download
^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: Asymmetric partition-wise JOIN
2019-08-12 06:03 Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-08-22 16:05 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-08-23 22:01 ` Re: Asymmetric partition-wise JOIN Thomas Munro <[email protected]>
2019-08-24 08:33 ` Re: Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
@ 2020-08-26 13:32 ` Amul Sul <[email protected]>
1 sibling, 0 replies; 28+ messages in thread
From: Amul Sul @ 2020-08-26 13:32 UTC (permalink / raw)
To: Kohei KaiGai <[email protected]>; +Cc: Thomas Munro <[email protected]>; pgsql-hackers
On Sat, Aug 24, 2019 at 2:03 PM Kohei KaiGai <[email protected]> wrote:
>
> 2019年8月24日(土) 7:02 Thomas Munro <[email protected]>:
> >
> > On Fri, Aug 23, 2019 at 4:05 AM Kohei KaiGai <[email protected]> wrote:
> > > We can consider the table join ptable X t1 above is equivalent to:
> > > (ptable_p0 + ptable_p1 + ptable_p2) X t1
> > > = (ptable_p0 X t1) + (ptable_p1 X t1) + (ptable_p2 X t1)
> > > It returns an equivalent result, however, rows are already reduced by HashJoin
> > > in the individual leaf of Append, so CPU-cycles consumed by Append node can
> > > be cheaper.
> > >
> > > On the other hands, it has a downside because t1 must be read 3 times and
> > > hash table also must be built 3 times. It increases the expected cost,
> > > so planner
> > > may not choose the asymmetric partition-wise join plan.
> >
> > What if you include the partition constraint as a filter on t1? So you get:
> >
> > ptable X t1 =
> > (ptable_p0 X (σ hash(dist)%4=0 (t1))) +
> > (ptable_p1 X (σ hash(dist)%4=1 (t1))) +
> > (ptable_p2 X (σ hash(dist)%4=2 (t1))) +
> > (ptable_p3 X (σ hash(dist)%4=3 (t1)))
> >
> > Pros:
> > 1. The hash tables will not contain unnecessary junk.
> > 2. You'll get the right answer if t1 is on the outer side of an outer join.
> > 3. If this runs underneath a Parallel Append and t1 is big enough
> > then workers will hopefully cooperate and do a synchronised scan of
> > t1.
> > 4. The filter might enable a selective and efficient plan like an index scan.
> >
> > Cons:
> > 1. The filter might not enable a selective and efficient plan, and
> > therefore cause extra work.
> >
> > (It's a little weird in this example because don't usually see hash
> > functions in WHERE clauses, but that could just as easily be dist
> > BETWEEN 1 AND 99 or any other partition constraint.)
> >
> It requires the join-key must include the partition key and also must be
> equality-join, doesn't it?
> If ptable and t1 are joined using ptable.dist = t1.foo, we can distribute
> t1 for each leaf table with "WHERE hash(foo)%4 = xxx" according to
> the partition bounds, indeed.
>
> In case when some of partition leafs are pruned, it is exactly beneficial
> because relevant rows to be referenced by the pruned child relations
> are waste of memory.
>
> On the other hands, it eventually consumes almost equivalent amount
> of memory to load the inner relations, if no leafs are pruned, and if we
> could extend the Hash-node to share the hash-table with sibling join-nodess.
>
> > > One idea I have is, sibling HashJoin shares a hash table that was built once
> > > by any of the sibling Hash plan. Right now, it is not implemented yet.
> >
> > Yeah, I've thought a little bit about that in the context of Parallel
> > Repartition. I'm interested in combining intra-node partitioning
> > (where a single plan node repartitions data among workers on the fly)
> > with inter-node partitioning (like PWJ, where partitions are handled
> > by different parts of the plan, considered at planning time); you
> > finish up needing to have nodes in the plan that 'receive' tuples for
> > each partition, to match up with the PWJ plan structure. That's not
> > entirely unlike CTE references, and not entirely unlike your idea of
> > somehow sharing the same hash table. I ran into a number of problems
> > while thinking about that, which I should write about in another
> > thread.
> >
> Hmm. Do you intend the inner-path may have different behavior according
> to the partition bounds definition where the outer-path to be joined?
> Let me investigate its pros & cons.
>
> The reasons why I think the idea of sharing the same hash table is reasonable
> in this scenario are:
> 1. We can easily extend the idea for parallel optimization. A hash table on DSM
> segment, once built, can be shared by all the siblings in all the
> parallel workers.
> 2. We can save the memory consumption regardless of the join-keys and
> partition-keys, even if these are not involved in the query.
>
> On the other hands, below are the downside. Potentially, combined use of
> your idea may help these cases:
> 3. Distributed inner-relation cannot be outer side of XXX OUTER JOIN.
> 4. Hash table contains rows to be referenced by only pruned partition leafs.
>
+ many, for the sharable hash of the inner table of the join. IMHO,
this could be the most interesting and captivating thing about this feature.
But might be a complicated piece, is that still on the plan?
Regards,
Amul
^ permalink raw reply [nested|flat] 28+ messages in thread
end of thread, other threads:[~2021-09-15 06:31 UTC | newest]
Thread overview: 28+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2019-08-12 06:03 Asymmetric partition-wise JOIN Kohei KaiGai <[email protected]>
2019-08-22 16:05 ` Kohei KaiGai <[email protected]>
2019-08-23 22:01 ` Thomas Munro <[email protected]>
2019-08-24 08:33 ` Kohei KaiGai <[email protected]>
2019-12-01 03:24 ` Michael Paquier <[email protected]>
2019-12-27 07:34 ` Kohei KaiGai <[email protected]>
2020-03-27 14:44 ` David Steele <[email protected]>
2020-07-01 09:10 ` Daniel Gustafsson <[email protected]>
2020-08-21 06:02 ` Andrey V. Lepikhov <[email protected]>
2020-08-25 09:12 ` Daniel Gustafsson <[email protected]>
2020-11-09 10:53 ` Anastasia Lubennikova <[email protected]>
2020-11-30 14:43 ` Anastasia Lubennikova <[email protected]>
2021-04-09 11:14 ` Andrey V. Lepikhov <[email protected]>
2021-04-30 03:10 ` Andrey V. Lepikhov <[email protected]>
2021-05-27 04:27 ` Andrey Lepikhov <[email protected]>
2021-06-18 12:02 ` Alexander Pyhalov <[email protected]>
2021-07-05 09:57 ` Andrey Lepikhov <[email protected]>
2021-07-05 20:15 ` Zhihong Yu <[email protected]>
2021-07-06 09:28 ` Andrey Lepikhov <[email protected]>
2021-07-06 13:09 ` Alexander Pyhalov <[email protected]>
2021-07-15 06:32 ` Andrey Lepikhov <[email protected]>
2021-07-15 13:16 ` Ibrar Ahmed <[email protected]>
2021-09-09 09:50 ` Aleksander Alekseev <[email protected]>
2021-09-09 15:38 ` Jaime Casanova <[email protected]>
2021-09-14 06:37 ` Andrey V. Lepikhov <[email protected]>
2021-09-15 06:31 ` Andrey Lepikhov <[email protected]>
2020-07-06 08:46 ` Andrey V. Lepikhov <[email protected]>
2020-08-26 13:32 ` Amul Sul <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox