From: Jesper Pedersen <jesper.pedersen@redhat.com>
To: Amit Langote <Langote_Amit_f8@lab.ntt.co.jp>
Cc: Imai, Yoshikazu <imai.yoshikazu@jp.fujitsu.com>
Cc: Amit Langote <amitlangote09@gmail.com>
Cc: Tom Lane <tgl@sss.pgh.pa.us>
Cc: David Rowley <david.rowley@2ndquadrant.com>
Cc: Alvaro Herrera <alvherre@2ndquadrant.com>
Cc: Justin Pryzby <pryzby@telsasoft.com>
Cc: Pg Hackers <pgsql-hackers@postgresql.org>
Subject: Re: speeding up planning with partitions
Date: Tue, 12 Mar 2019 09:23:25 -0400
Message-ID: <d768d90d-628e-6cdc-513d-f50a0ce02cea@redhat.com> (raw)
In-Reply-To: <af3da198-e4aa-0535-8b13-40b02a4a7c41@lab.ntt.co.jp>
References: <CAKJS1f9XAePd2+8F4=wtHKdDdckgKcSoXb-ZbU8Wbt86mhLxKA@mail.gmail.com>
<b8a139b8-2f5d-6079-67b2-99957af3fc36@lab.ntt.co.jp>
<7453.1550518972@sss.pgh.pa.us>
<0a70234e-b2e6-debc-2fb5-e8fd927caa00@lab.ntt.co.jp>
<1002.1550596071@sss.pgh.pa.us>
<30961.1550609873@sss.pgh.pa.us>
<250c61bc-479c-92f4-d19d-d768e3bcdcbc@lab.ntt.co.jp>
<13510.1550677844@sss.pgh.pa.us>
<a47e6214-bd46-ab4b-2f84-f9f726e04642@lab.ntt.co.jp>
<35b02675-83c2-a12b-961a-fec249b06f35@lab.ntt.co.jp>
<20190222165932.GG28750@telsasoft.com>
<CA+HiwqEUwfg4s=4=is-+7k-WM-2Gs9K3iFB96QNm_eafcBozqQ@mail.gmail.com>
<0F97FA9ABBDBE54F91744A9B37151A51293422@g01jpexmbkw24>
<8098beec-fc04-303d-55b9-73c3d94ba1ea@lab.ntt.co.jp>
<b5362c29-cca1-de7b-13d0-9e51f9911185@lab.ntt.co.jp>
<80afef3f-1870-5cb3-eeaf-7d8ecb41994c@lab.ntt.co.jp>
<af3da198-e4aa-0535-8b13-40b02a4a7c41@lab.ntt.co.jp>
Hi Amit,
On 3/12/19 4:22 AM, Amit Langote wrote:
> I wrestled with this idea a bit and concluded that we don't have to
> postpone *all* of preprocess_targetlist() processing to query_planner,
> only the part that adds row mark "junk" Vars, because only those matter
> for the problem being solved. To recap, the problem is that delaying
> adding inheritance children (and hence their row marks if any) means we
> can't add "junk" columns needed to implement row marks, because which ones
> to add is not clear until we've seen all the children.
>
> I propose that we delay the addition of "junk" Vars to query_planner() so
> that it doesn't stand in the way of deferring inheritance expansion to
> query_planner() too. That means the order of reltarget expressions for
> row-marked rels will change, but I assume that's OK. At least it will be
> consistent for both non-inherited baserels and inherited ones.
>
> Attached updated version of the patches with the above proposal
> implemented by patch 0002. To summarize, the patches are as follows:
>
> 0001: make building of "other rels" a separate step that runs after
> deconstruct_jointree(), implemented by a new subroutine of query_planner
> named add_other_rels_to_query()
>
> 0002: delay adding "junk" Vars to after add_other_rels_to_query()
>
> 0003: delay inheritance expansion to add_other_rels_to_query()
>
> 0004, 0005: adjust inheritance_planner() to account for the fact that
> inheritance is now expanded by query_planner(), not subquery_planner()
>
> 0006: perform partition pruning while inheritance is being expanded,
> instead of during set_append_append_rel_size()
>
> 0007: add a 'live_parts' field to RelOptInfo to store partition indexes
> (not RT indexes) of unpruned partitions, which speeds up looping over
> part_rels array of the partitioned parent
>
> 0008: avoid copying PartitionBoundInfo struct for planning
>
After applying 0004 check-world fails with the attached. CFBot agrees [1].
[1] https://travis-ci.org/postgresql-cfbot/postgresql/builds/505107884
Best regards,
Jesper
diff -U3 /home/jpedersen/PostgreSQL/postgresql/contrib/postgres_fdw/expected/postgres_fdw.out /home/jpedersen/PostgreSQL/postgresql/contrib/postgres_fdw/results/postgres_fdw.out
--- /home/jpedersen/PostgreSQL/postgresql/contrib/postgres_fdw/expected/postgres_fdw.out 2019-03-12 07:46:08.430690229 -0400
+++ /home/jpedersen/PostgreSQL/postgresql/contrib/postgres_fdw/results/postgres_fdw.out 2019-03-12 07:59:01.134285159 -0400
@@ -7190,8 +7190,8 @@
-- Check UPDATE with inherited target and an inherited source table
explain (verbose, costs off)
update bar set f2 = f2 + 100 where f1 in (select f1 from foo);
- QUERY PLAN
----------------------------------------------------------------------------------------------
+ QUERY PLAN
+---------------------------------------------------------------------------------------
Update on public.bar
Update on public.bar
Foreign Update on public.bar2
@@ -7214,22 +7214,22 @@
Output: foo2.f1, foo2.ctid, foo2.*, foo2.tableoid
Remote SQL: SELECT f1, f2, f3, ctid FROM public.loct1
-> Hash Join
- Output: bar2.f1, (bar2.f2 + 100), bar2.f3, bar2.ctid, foo.ctid, foo.*, foo.tableoid
+ Output: bar2.f1, (bar2.f2 + 100), bar2.f3, bar2.ctid, foo.ctid, foo.*
Inner Unique: true
Hash Cond: (bar2.f1 = foo.f1)
-> Foreign Scan on public.bar2
Output: bar2.f1, bar2.f2, bar2.f3, bar2.ctid
Remote SQL: SELECT f1, f2, f3, ctid FROM public.loct2 FOR UPDATE
-> Hash
- Output: foo.f1, foo.ctid, foo.*, foo.tableoid
+ Output: foo.f1, foo.ctid, foo.*
-> HashAggregate
- Output: foo.f1, foo.ctid, foo.*, foo.tableoid
+ Output: foo.f1, foo.ctid, foo.*
Group Key: foo.f1
-> Append
-> Seq Scan on public.foo
- Output: foo.f1, foo.ctid, foo.*, foo.tableoid
+ Output: foo.f1, foo.ctid, foo.*
-> Foreign Scan on public.foo2
- Output: foo2.f1, foo2.ctid, foo2.*, foo2.tableoid
+ Output: foo2.f1, foo2.ctid, foo2.*
Remote SQL: SELECT f1, f2, f3, ctid FROM public.loct1
(39 rows)
Attachments:
[text/plain] regression.diff.txt (2.5K, ../d768d90d-628e-6cdc-513d-f50a0ce02cea@redhat.com/2-regression.diff.txt)
download | inline:
diff -U3 /home/jpedersen/PostgreSQL/postgresql/contrib/postgres_fdw/expected/postgres_fdw.out /home/jpedersen/PostgreSQL/postgresql/contrib/postgres_fdw/results/postgres_fdw.out
--- /home/jpedersen/PostgreSQL/postgresql/contrib/postgres_fdw/expected/postgres_fdw.out 2019-03-12 07:46:08.430690229 -0400
+++ /home/jpedersen/PostgreSQL/postgresql/contrib/postgres_fdw/results/postgres_fdw.out 2019-03-12 07:59:01.134285159 -0400
@@ -7190,8 +7190,8 @@
-- Check UPDATE with inherited target and an inherited source table
explain (verbose, costs off)
update bar set f2 = f2 + 100 where f1 in (select f1 from foo);
- QUERY PLAN
----------------------------------------------------------------------------------------------
+ QUERY PLAN
+---------------------------------------------------------------------------------------
Update on public.bar
Update on public.bar
Foreign Update on public.bar2
@@ -7214,22 +7214,22 @@
Output: foo2.f1, foo2.ctid, foo2.*, foo2.tableoid
Remote SQL: SELECT f1, f2, f3, ctid FROM public.loct1
-> Hash Join
- Output: bar2.f1, (bar2.f2 + 100), bar2.f3, bar2.ctid, foo.ctid, foo.*, foo.tableoid
+ Output: bar2.f1, (bar2.f2 + 100), bar2.f3, bar2.ctid, foo.ctid, foo.*
Inner Unique: true
Hash Cond: (bar2.f1 = foo.f1)
-> Foreign Scan on public.bar2
Output: bar2.f1, bar2.f2, bar2.f3, bar2.ctid
Remote SQL: SELECT f1, f2, f3, ctid FROM public.loct2 FOR UPDATE
-> Hash
- Output: foo.f1, foo.ctid, foo.*, foo.tableoid
+ Output: foo.f1, foo.ctid, foo.*
-> HashAggregate
- Output: foo.f1, foo.ctid, foo.*, foo.tableoid
+ Output: foo.f1, foo.ctid, foo.*
Group Key: foo.f1
-> Append
-> Seq Scan on public.foo
- Output: foo.f1, foo.ctid, foo.*, foo.tableoid
+ Output: foo.f1, foo.ctid, foo.*
-> Foreign Scan on public.foo2
- Output: foo2.f1, foo2.ctid, foo2.*, foo2.tableoid
+ Output: foo2.f1, foo2.ctid, foo2.*
Remote SQL: SELECT f1, f2, f3, ctid FROM public.loct1
(39 rows)
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: jesper.pedersen@redhat.com, Langote_Amit_f8@lab.ntt.co.jp, imai.yoshikazu@jp.fujitsu.com, amitlangote09@gmail.com, tgl@sss.pgh.pa.us, david.rowley@2ndquadrant.com, alvherre@2ndquadrant.com, pryzby@telsasoft.com
Subject: Re: speeding up planning with partitions
In-Reply-To: <d768d90d-628e-6cdc-513d-f50a0ce02cea@redhat.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox