pg.ddx.io pgsql-hackers@postgresql.org mailing list archive
help / color / mirror / Atom feedEarly WIP/PoC for inlining CTEs
112+ messages / 25 participants
[nested] [flat]
* Early WIP/PoC for inlining CTEs
@ 2018-07-24 22:28 Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-24 23:03 ` Re: Early WIP/PoC for inlining CTEs Jeremy Finzel <finzelj@gmail.com>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
0 siblings, 2 replies; 112+ messages in thread
From: Andrew Gierth @ 2018-07-24 22:28 UTC (permalink / raw)
To: pgsql-hackers
About a year ago I was briefly in discussion/collaboration with Adam Sah
regarding the topic of inlining CTEs into the query rather than treating
them as optimization barriers. We didn't take it very far (he sent me
some stuff, I wrote some stuff and sent it back, things kind of got
dropped at that point); but there's been some recent discussion of this
and some people have expressed an interest in seeing the code.
So I'm posting the parts that I wrote for the benefit of anyone wanting
to pick up the issue again. The assumption of this code is that some
form of syntax would exist to mark materialized CTEs and set the
"ctematerialized" flag.
I haven't rebased this or tested it since last year; this patch is
against b81eba6a65.
Posted for discussion, further development, criticism, whatever; feel
free to include this (with credit) in any relevant patch. Consider this
released under the PG license.
--
Andrew (irc:RhodiumToad)
Attachments:
[text/x-patch] cteinline.patch (7.7K, ../../87sh48ffhb.fsf@news-spur.riddles.org.uk/2-cteinline.patch)
download | inline diff:
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index c1a83ca909..393c673164 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2474,6 +2474,7 @@ _copyCommonTableExpr(const CommonTableExpr *from)
COPY_NODE_FIELD(ctequery);
COPY_LOCATION_FIELD(location);
COPY_SCALAR_FIELD(cterecursive);
+ COPY_SCALAR_FIELD(ctematerialized);
COPY_SCALAR_FIELD(cterefcount);
COPY_NODE_FIELD(ctecolnames);
COPY_NODE_FIELD(ctecoltypes);
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index 7a700018e7..2112560871 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2781,6 +2781,7 @@ _equalCommonTableExpr(const CommonTableExpr *a, const CommonTableExpr *b)
COMPARE_NODE_FIELD(ctequery);
COMPARE_LOCATION_FIELD(location);
COMPARE_SCALAR_FIELD(cterecursive);
+ COMPARE_SCALAR_FIELD(ctematerialized);
COMPARE_SCALAR_FIELD(cterefcount);
COMPARE_NODE_FIELD(ctecolnames);
COMPARE_NODE_FIELD(ctecoltypes);
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 43d62062bc..2df4e6300a 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -3010,6 +3010,7 @@ _outCommonTableExpr(StringInfo str, const CommonTableExpr *node)
WRITE_NODE_FIELD(ctequery);
WRITE_LOCATION_FIELD(location);
WRITE_BOOL_FIELD(cterecursive);
+ WRITE_BOOL_FIELD(ctematerialized);
WRITE_INT_FIELD(cterefcount);
WRITE_NODE_FIELD(ctecolnames);
WRITE_NODE_FIELD(ctecoltypes);
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index ccb6a1f4ac..4705ea19d7 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -404,6 +404,7 @@ _readCommonTableExpr(void)
READ_NODE_FIELD(ctequery);
READ_LOCATION_FIELD(location);
READ_BOOL_FIELD(cterecursive);
+ READ_BOOL_FIELD(ctematerialized);
READ_INT_FIELD(cterefcount);
READ_NODE_FIELD(ctecolnames);
READ_NODE_FIELD(ctecoltypes);
diff --git a/src/backend/optimizer/plan/subselect.c b/src/backend/optimizer/plan/subselect.c
index 1103984779..63e5828ef9 100644
--- a/src/backend/optimizer/plan/subselect.c
+++ b/src/backend/optimizer/plan/subselect.c
@@ -1129,6 +1129,141 @@ hash_ok_operator(OpExpr *expr)
}
+struct inline_cte_walker_ctx
+{
+ const char *ctename;
+ int levelsup;
+ int refcount;
+ Query *ctequery;
+ CommonTableExpr *cte;
+};
+
+static bool inline_cte_walker(Node *node, void *ctxp)
+{
+ struct inline_cte_walker_ctx *ctx = ctxp;
+
+ if (!node)
+ return false;
+
+ if (IsA(node, Query))
+ {
+ /*
+ * This one is a bit tricky. It's our job to handle the recursion here,
+ * but we do some of the lifting normally handled by query_tree_walker
+ * in order to get the sequence of operations right.
+ *
+ * First, if the Query we're looking at is the one containing our CTE
+ * definition, then we don't need to recurse into our own CTE or CTEs
+ * that are earlier in the list than ours (since cteList has been
+ * sorted for us into dependency order). We could check whether a
+ * nested query is hiding ours, but that seems too much of an edge case
+ * to be worth optimizing (the levelsup check will ensure we don't
+ * replace any CTEs improperly). So we scan the cteList ourselves
+ * rather than having query_tree_walker do it.
+ *
+ * Second, we want to walk the rangetable _before_ replacing any
+ * RTE_CTE nodes, in order to avoid re-walking the subquery we just
+ * inlined. (range_table_walker, if told to visit the RTE nodes at all,
+ * visits them before their content.) So we have range_table_walker
+ * ignore the RTE nodes themselves and only walk their contents.
+ *
+ * Third, we scan the rangetable for RTE_CTE nodes to replace.
+ *
+ * Fourth, we use query_tree_walker to find and walk the rest of the
+ * query, telling it to ignore the rangetable and CTEs.
+ *
+ * Note that ctx->levelsup is -1 on entry the first time, since we need
+ * the incremented value to be 0 when scanning the content of the query
+ * containing the definition.
+ */
+ Query *query = castNode(Query, node);
+ ListCell *lc;
+ bool do_replace = ctx->levelsup >= 0;
+
+ ctx->levelsup++;
+
+ foreach (lc, query->cteList)
+ {
+ CommonTableExpr *cte = lfirst_node(CommonTableExpr, lc);
+
+ if (!do_replace && strcmp(cte->ctename, ctx->ctename) == 0)
+ do_replace = true;
+ else if (do_replace)
+ inline_cte_walker(cte->ctequery, ctxp);
+ }
+
+ range_table_walker(query->rtable, inline_cte_walker, ctxp, 0);
+
+ foreach (lc, query->rtable)
+ {
+ RangeTblEntry *rte = lfirst_node(RangeTblEntry, lc);
+
+ if (rte->rtekind == RTE_CTE &&
+ strcmp(rte->ctename, ctx->ctename) == 0 &&
+ rte->ctelevelsup == ctx->levelsup)
+ {
+ Query *newquery = ctx->ctequery;
+
+ /*
+ * We need to do some work here that view rewrite does not, and
+ * in turn we do not do some work that view rewrite does.
+ *
+ * Firstly, views can't have outer references but CTEs can
+ * (especially in the case of CTEs referencing other CTEs), so
+ * we need to fix up all levelsup attributes inside the CTE
+ * query.
+ *
+ * Secondly, views (and explicit subqueries) currently have
+ * different behaviour w.r.t. SELECT FOR UPDATE than CTEs do. A
+ * FOR UPDATE clause is treated as extending into views and
+ * subqueries, but not into CTEs. We preserve this distinction
+ * by not trying to push rowmarks into the new subquery.
+ *
+ * We avoid copyObject if possible because subquery processing
+ * copies the query too.
+ */
+ if (ctx->levelsup > 0)
+ {
+ newquery = copyObject(newquery);
+ IncrementVarSublevelsUp((Node *) newquery, ctx->levelsup, 1);
+ }
+
+ /*
+ * Here's where we do the actual substitution.
+ */
+ rte->rtekind = RTE_SUBQUERY;
+ rte->subquery = newquery;
+ rte->security_barrier = false;
+
+ ctx->refcount--;
+ }
+ }
+
+ query_tree_walker(query, inline_cte_walker, ctxp,
+ QTW_IGNORE_RANGE_TABLE | QTW_IGNORE_CTE_SUBQUERIES);
+
+ ctx->levelsup--;
+
+ return false;
+ }
+
+ return expression_tree_walker(node, inline_cte_walker, ctxp);
+}
+
+static void inline_cte(PlannerInfo *root, CommonTableExpr *cte)
+{
+ struct inline_cte_walker_ctx ctx;
+ ctx.ctequery = castNode(Query, cte->ctequery);
+ ctx.ctename = cte->ctename;
+ ctx.refcount = cte->cterefcount;
+ ctx.levelsup = -1;
+ ctx.cte = cte;
+ inline_cte_walker((Node *) root->parse, &ctx);
+ /* we must replace all references */
+ Assert(ctx.refcount == 0);
+}
+
+
/*
* SS_process_ctes: process a query's WITH list
*
@@ -1167,6 +1302,23 @@ SS_process_ctes(PlannerInfo *root)
}
/*
+ * Consider inlining the CTE rather than planning it separately.
+ *
+ * XXX this likely needs some additional checks.
+ */
+ if (cmdType == CMD_SELECT &&
+ !cte->ctematerialized &&
+ !cte->cterecursive &&
+ (castNode(Query, cte->ctequery)->rowMarks == NIL))
+ {
+ inline_cte(root, cte);
+
+ /* Make a dummy entry in cte_plan_ids */
+ root->cte_plan_ids = lappend_int(root->cte_plan_ids, -1);
+ continue;
+ }
+
+ /*
* Copy the source Query node. Probably not necessary, but let's keep
* this similar to make_subplan.
*/
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 732e5d6788..901a3a295f 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -1372,6 +1372,7 @@ typedef struct CommonTableExpr
int location; /* token location, or -1 if unknown */
/* These fields are set during parse analysis: */
bool cterecursive; /* is this CTE actually recursive? */
+ bool ctematerialized; /* is this an optimization fence? */
int cterefcount; /* number of RTEs referencing this CTE
* (excluding internal self-references) */
List *ctecolnames; /* list of output column names */
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
@ 2018-07-24 23:03 ` Jeremy Finzel <finzelj@gmail.com>
2018-07-24 23:10 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 112+ messages in thread
From: Jeremy Finzel @ 2018-07-24 23:03 UTC (permalink / raw)
To: Andrew Gierth <andrew@tao11.riddles.org.uk>; +Cc: pgsql-hackers
On Tue, Jul 24, 2018 at 5:28 PM Andrew Gierth <andrew@tao11.riddles.org.uk>
wrote:
> About a year ago I was briefly in discussion/collaboration with Adam Sah
> regarding the topic of inlining CTEs into the query rather than treating
> them as optimization barriers. We didn't take it very far (he sent me
> some stuff, I wrote some stuff and sent it back, things kind of got
> dropped at that point); but there's been some recent discussion of this
> and some people have expressed an interest in seeing the code.
>
> So I'm posting the parts that I wrote for the benefit of anyone wanting
> to pick up the issue again. The assumption of this code is that some
> form of syntax would exist to mark materialized CTEs and set the
> "ctematerialized" flag.
>
> I haven't rebased this or tested it since last year; this patch is
> against b81eba6a65.
>
> Posted for discussion, further development, criticism, whatever; feel
> free to include this (with credit) in any relevant patch. Consider this
> released under the PG license.
>
> --
> Andrew (irc:RhodiumToad)
>
> In our environment we often want this to be a fence. For example it can
be used to only have smaller numbers of joins in each cte and not hit the
join collapse limit, or when we really know more about the subquery than
the optimizer and have something really specific there . So in general I
would not want the default functionality to change all of the queries we
have already written with this in mind. I do however like the idea of this
feature being an option, but I would question whether it perhaps worked the
other way around where you have to mark a CTE as not being a fence.
Curious what other RDBMSs do here?
Thanks,
Jeremy
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-24 23:03 ` Re: Early WIP/PoC for inlining CTEs Jeremy Finzel <finzelj@gmail.com>
@ 2018-07-24 23:10 ` Andres Freund <andres@anarazel.de>
2018-07-24 23:48 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-24 23:49 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-07-24 23:50 ` Re: Early WIP/PoC for inlining CTEs Gavin Flower <GavinFlower@archidevsys.co.nz>
2019-01-11 21:25 ` Re: Early WIP/PoC for inlining CTEs Merlin Moncure <mmoncure@gmail.com>
0 siblings, 4 replies; 112+ messages in thread
From: Andres Freund @ 2018-07-24 23:10 UTC (permalink / raw)
To: Jeremy Finzel <finzelj@gmail.com>; +Cc: Andrew Gierth <andrew@tao11.riddles.org.uk>; pgsql-hackers
On 2018-07-24 18:03:43 -0500, Jeremy Finzel wrote:
> On Tue, Jul 24, 2018 at 5:28 PM Andrew Gierth <andrew@tao11.riddles.org.uk>
> wrote:
>
> > About a year ago I was briefly in discussion/collaboration with Adam Sah
> > regarding the topic of inlining CTEs into the query rather than treating
> > them as optimization barriers. We didn't take it very far (he sent me
> > some stuff, I wrote some stuff and sent it back, things kind of got
> > dropped at that point); but there's been some recent discussion of this
> > and some people have expressed an interest in seeing the code.
> >
> > So I'm posting the parts that I wrote for the benefit of anyone wanting
> > to pick up the issue again. The assumption of this code is that some
> > form of syntax would exist to mark materialized CTEs and set the
> > "ctematerialized" flag.
> >
> > I haven't rebased this or tested it since last year; this patch is
> > against b81eba6a65.
> >
> > Posted for discussion, further development, criticism, whatever; feel
> > free to include this (with credit) in any relevant patch. Consider this
> > released under the PG license.
> >
> > --
> > Andrew (irc:RhodiumToad)
> >
> > In our environment we often want this to be a fence. For example it can
> be used to only have smaller numbers of joins in each cte and not hit the
> join collapse limit, or when we really know more about the subquery than
> the optimizer and have something really specific there . So in general I
> would not want the default functionality to change all of the queries we
> have already written with this in mind. I do however like the idea of this
> feature being an option, but I would question whether it perhaps worked the
> other way around where you have to mark a CTE as not being a fence.
This essentially has been discussed already:
http://archives.postgresql.org/message-id/5351711493487900%40web53g.yandex.ru
My read of the concensus (in which I am in the majority, so I might be
biased) is that we do want inlining to be the default. We were thinking
that it'd be necessary to provide a way to force inlining on the SQL
level for individual CTEs.
> Curious what other RDBMSs do here?
They largely inline by default.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-24 23:03 ` Re: Early WIP/PoC for inlining CTEs Jeremy Finzel <finzelj@gmail.com>
2018-07-24 23:10 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
@ 2018-07-24 23:48 ` Andrew Gierth <andrew@tao11.riddles.org.uk>
3 siblings, 0 replies; 112+ messages in thread
From: Andrew Gierth @ 2018-07-24 23:48 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Jeremy Finzel <finzelj@gmail.com>; pgsql-hackers
>>>>> "Andres" == Andres Freund <andres@anarazel.de> writes:
Andres> My read of the concensus (in which I am in the majority, so I
Andres> might be biased) is that we do want inlining to be the default.
Andres> We were thinking that it'd be necessary to provide a way to
Andres> force inlining on the SQL level for individual CTEs.
For the record, here is where it came up in the original discussion
exactly 10 years ago when the feature was being added:
https://www.postgresql.org/message-id/flat/87myk1rg4z.fsf%40news-spur.riddles.org.uk#35efe8ef3705181...
--
Andrew (irc:RhodiumToad)
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-24 23:03 ` Re: Early WIP/PoC for inlining CTEs Jeremy Finzel <finzelj@gmail.com>
2018-07-24 23:10 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
@ 2018-07-24 23:49 ` Tom Lane <tgl@sss.pgh.pa.us>
2018-07-24 23:54 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2018-07-25 00:04 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
3 siblings, 2 replies; 112+ messages in thread
From: Tom Lane @ 2018-07-24 23:49 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Jeremy Finzel <finzelj@gmail.com>; Andrew Gierth <andrew@tao11.riddles.org.uk>; pgsql-hackers
Andres Freund <andres@anarazel.de> writes:
> On 2018-07-24 18:03:43 -0500, Jeremy Finzel wrote:
>> On Tue, Jul 24, 2018 at 5:28 PM Andrew Gierth <andrew@tao11.riddles.org.uk>
>> wrote:
>>> Posted for discussion, further development, criticism, whatever; feel
>>> free to include this (with credit) in any relevant patch. Consider this
>>> released under the PG license.
>> In our environment we often want this to be a fence. For example it can
>> be used to only have smaller numbers of joins in each cte and not hit the
>> join collapse limit, or when we really know more about the subquery than
>> the optimizer and have something really specific there . So in general I
>> would not want the default functionality to change all of the queries we
>> have already written with this in mind. I do however like the idea of this
>> feature being an option, but I would question whether it perhaps worked the
>> other way around where you have to mark a CTE as not being a fence.
> This essentially has been discussed already:
> http://archives.postgresql.org/message-id/5351711493487900%40web53g.yandex.ru
> My read of the concensus (in which I am in the majority, so I might be
> biased) is that we do want inlining to be the default. We were thinking
> that it'd be necessary to provide a way to force inlining on the SQL
> level for individual CTEs.
We can't inline wCTEs (those containing insert/update/delete) without
risk of semantics change. I'd also not favor changing the semantics
for CTEs that are read more than once by the parent query. However,
a singly-referenced SELECT CTE could reasonably be treated as equivalent
to a sub-select-in-FROM, and then you would have the same mechanisms
for preventing inlining as you do for those cases, e.g. OFFSET 0.
And sticking in OFFSET 0 would be backwards-compatible too: your
code would still work the same in older releases, unlike if we invent
new syntax for this.
So if we're going to go in this direction, that's pretty much how
I'd envision it working.
>> Curious what other RDBMSs do here?
> They largely inline by default.
Even for multi-referenced CTEs?
regards, tom lane
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-24 23:03 ` Re: Early WIP/PoC for inlining CTEs Jeremy Finzel <finzelj@gmail.com>
2018-07-24 23:10 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2018-07-24 23:49 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
@ 2018-07-24 23:54 ` Andres Freund <andres@anarazel.de>
2018-07-24 23:57 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 1 reply; 112+ messages in thread
From: Andres Freund @ 2018-07-24 23:54 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Jeremy Finzel <finzelj@gmail.com>; Andrew Gierth <andrew@tao11.riddles.org.uk>; pgsql-hackers
Hi,
On 2018-07-24 19:49:19 -0400, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > On 2018-07-24 18:03:43 -0500, Jeremy Finzel wrote:
> >> On Tue, Jul 24, 2018 at 5:28 PM Andrew Gierth <andrew@tao11.riddles.org.uk>
> >> wrote:
> >>> Posted for discussion, further development, criticism, whatever; feel
> >>> free to include this (with credit) in any relevant patch. Consider this
> >>> released under the PG license.
>
> >> In our environment we often want this to be a fence. For example it can
> >> be used to only have smaller numbers of joins in each cte and not hit the
> >> join collapse limit, or when we really know more about the subquery than
> >> the optimizer and have something really specific there . So in general I
> >> would not want the default functionality to change all of the queries we
> >> have already written with this in mind. I do however like the idea of this
> >> feature being an option, but I would question whether it perhaps worked the
> >> other way around where you have to mark a CTE as not being a fence.
>
> > This essentially has been discussed already:
> > http://archives.postgresql.org/message-id/5351711493487900%40web53g.yandex.ru
> > My read of the concensus (in which I am in the majority, so I might be
> > biased) is that we do want inlining to be the default. We were thinking
> > that it'd be necessary to provide a way to force inlining on the SQL
> > level for individual CTEs.
>
> We can't inline wCTEs (those containing insert/update/delete) without
> risk of semantics change.
Right.
> I'd also not favor changing the semantics for CTEs that are read more
> than once by the parent query.
I think medium term it'd be good to do a cost based analysis for that. I
think it's fine to not do that immediately, but we should imo keep that
in mind.
> However, a singly-referenced SELECT CTE could reasonably be treated as
> equivalent to a sub-select-in-FROM, and then you would have the same
> mechanisms for preventing inlining as you do for those cases,
> e.g. OFFSET 0. And sticking in OFFSET 0 would be backwards-compatible
> too: your code would still work the same in older releases, unlike if
> we invent new syntax for this.
I still think this is just doubling down on prior mistakes.
> >> Curious what other RDBMSs do here?
>
> > They largely inline by default.
>
> Even for multi-referenced CTEs?
I don't know.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-24 23:03 ` Re: Early WIP/PoC for inlining CTEs Jeremy Finzel <finzelj@gmail.com>
2018-07-24 23:10 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2018-07-24 23:49 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-07-24 23:54 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
@ 2018-07-24 23:57 ` Tom Lane <tgl@sss.pgh.pa.us>
2018-07-25 00:04 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2018-07-25 18:39 ` Re: Early WIP/PoC for inlining CTEs Nico Williams <nico@cryptonector.com>
0 siblings, 2 replies; 112+ messages in thread
From: Tom Lane @ 2018-07-24 23:57 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Jeremy Finzel <finzelj@gmail.com>; Andrew Gierth <andrew@tao11.riddles.org.uk>; pgsql-hackers
Andres Freund <andres@anarazel.de> writes:
> On 2018-07-24 19:49:19 -0400, Tom Lane wrote:
>> However, a singly-referenced SELECT CTE could reasonably be treated as
>> equivalent to a sub-select-in-FROM, and then you would have the same
>> mechanisms for preventing inlining as you do for those cases,
>> e.g. OFFSET 0. And sticking in OFFSET 0 would be backwards-compatible
>> too: your code would still work the same in older releases, unlike if
>> we invent new syntax for this.
> I still think this is just doubling down on prior mistakes.
Not following what you think a better alternative is? I'd be the
first to agree that OFFSET 0 is a hack, but people are used to it.
Assuming that we go for inline-by-default for at least some cases,
there's a separate discussion to be had about whether it's worth
making a planner-control GUC to force the old behavior. I'm not
very excited about that, but I bet some people will be.
regards, tom lane
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-24 23:03 ` Re: Early WIP/PoC for inlining CTEs Jeremy Finzel <finzelj@gmail.com>
2018-07-24 23:10 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2018-07-24 23:49 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-07-24 23:54 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2018-07-24 23:57 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
@ 2018-07-25 00:04 ` Andres Freund <andres@anarazel.de>
2018-07-25 00:08 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
1 sibling, 1 reply; 112+ messages in thread
From: Andres Freund @ 2018-07-25 00:04 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Jeremy Finzel <finzelj@gmail.com>; Andrew Gierth <andrew@tao11.riddles.org.uk>; pgsql-hackers
Hi,
On 2018-07-24 19:57:49 -0400, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > On 2018-07-24 19:49:19 -0400, Tom Lane wrote:
> >> However, a singly-referenced SELECT CTE could reasonably be treated as
> >> equivalent to a sub-select-in-FROM, and then you would have the same
> >> mechanisms for preventing inlining as you do for those cases,
> >> e.g. OFFSET 0. And sticking in OFFSET 0 would be backwards-compatible
> >> too: your code would still work the same in older releases, unlike if
> >> we invent new syntax for this.
>
> > I still think this is just doubling down on prior mistakes.
>
> Not following what you think a better alternative is?
An explicit keyword controlling the behaviour. WITH ... foo AS
MATERIALIZED (query) or whatnot.
> I'd be the first to agree that OFFSET 0 is a hack, but people are used
> to it.
So are they to CTEs being blocking.
Consider e.g. the case of being able to push down constraints into CTEs
(which have been inlined to allow for that). Even in queries with a
non-0 OFFSET you can push down in a number of cases, making CTE inlining
useful. You can't tack on an OFFSET 0 controlling that, without going
for a superflous subquery that just has an OFFSET 0, which is fairly
ridiculous. What if we learn to inline subqueries with some offsets?
> Assuming that we go for inline-by-default for at least some cases,
> there's a separate discussion to be had about whether it's worth
> making a planner-control GUC to force the old behavior. I'm not
> very excited about that, but I bet some people will be.
Yea, I am not either. I think we shouldn't go for it.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-24 23:03 ` Re: Early WIP/PoC for inlining CTEs Jeremy Finzel <finzelj@gmail.com>
2018-07-24 23:10 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2018-07-24 23:49 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-07-24 23:54 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2018-07-24 23:57 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-07-25 00:04 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
@ 2018-07-25 00:08 ` Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 00:27 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 112+ messages in thread
From: Andrew Gierth @ 2018-07-25 00:08 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Jeremy Finzel <finzelj@gmail.com>; pgsql-hackers
>>>>> "Andres" == Andres Freund <andres@anarazel.de> writes:
Andres> Even in queries with a non-0 OFFSET you can push down in a
Andres> number of cases,
really?
--
Andrew (irc:RhodiumToad)
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-24 23:03 ` Re: Early WIP/PoC for inlining CTEs Jeremy Finzel <finzelj@gmail.com>
2018-07-24 23:10 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2018-07-24 23:49 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-07-24 23:54 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2018-07-24 23:57 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-07-25 00:04 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2018-07-25 00:08 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
@ 2018-07-25 00:27 ` Andres Freund <andres@anarazel.de>
0 siblings, 0 replies; 112+ messages in thread
From: Andres Freund @ 2018-07-25 00:27 UTC (permalink / raw)
To: Andrew Gierth <andrew@tao11.riddles.org.uk>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Jeremy Finzel <finzelj@gmail.com>; pgsql-hackers
On 2018-07-25 01:08:44 +0100, Andrew Gierth wrote:
> >>>>> "Andres" == Andres Freund <andres@anarazel.de> writes:
>
> Andres> Even in queries with a non-0 OFFSET you can push down in a
> Andres> number of cases,
>
> really?
Yea. I guess it's a bit dependant on what kind of behaviour you consider
as "pushing down". I'm doubtful it's worth the analytical complexity on
ensuring it's safe, however. With knowledge from the outer query you
e.g. can: trim the target list; remove outer joins below the OFFSET 0;
push down a restriction into an outer join below the OFFSET if that's
guaranteed to only return max one row, and not needed if not matching
the restrcition. I'm sure you can come up with more?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-24 23:03 ` Re: Early WIP/PoC for inlining CTEs Jeremy Finzel <finzelj@gmail.com>
2018-07-24 23:10 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2018-07-24 23:49 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-07-24 23:54 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2018-07-24 23:57 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
@ 2018-07-25 18:39 ` Nico Williams <nico@cryptonector.com>
1 sibling, 0 replies; 112+ messages in thread
From: Nico Williams @ 2018-07-25 18:39 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Andres Freund <andres@anarazel.de>; Jeremy Finzel <finzelj@gmail.com>; Andrew Gierth <andrew@tao11.riddles.org.uk>; pgsql-hackers
On Tue, Jul 24, 2018 at 07:57:49PM -0400, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > On 2018-07-24 19:49:19 -0400, Tom Lane wrote:
> >> However, a singly-referenced SELECT CTE could reasonably be treated as
> >> equivalent to a sub-select-in-FROM, and then you would have the same
> >> mechanisms for preventing inlining as you do for those cases,
> >> e.g. OFFSET 0. And sticking in OFFSET 0 would be backwards-compatible
> >> too: your code would still work the same in older releases, unlike if
> >> we invent new syntax for this.
>
> > I still think this is just doubling down on prior mistakes.
>
> Not following what you think a better alternative is? I'd be the
> first to agree that OFFSET 0 is a hack, but people are used to it.
>
> Assuming that we go for inline-by-default for at least some cases,
> there's a separate discussion to be had about whether it's worth
> making a planner-control GUC to force the old behavior. I'm not
> very excited about that, but I bet some people will be.
It is widely known that CTEs in PG are optimizer barriers.
That actually is useful, and I do make use of that fact (though I'm not
proud of it).
My proposal is that PG add an extension for specifying that a CTE is to
be materialized (barrier) or not (then inlined).
Nico
--
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-24 23:03 ` Re: Early WIP/PoC for inlining CTEs Jeremy Finzel <finzelj@gmail.com>
2018-07-24 23:10 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2018-07-24 23:49 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
@ 2018-07-25 00:04 ` Andrew Gierth <andrew@tao11.riddles.org.uk>
1 sibling, 0 replies; 112+ messages in thread
From: Andrew Gierth @ 2018-07-25 00:04 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Andres Freund <andres@anarazel.de>; Jeremy Finzel <finzelj@gmail.com>; pgsql-hackers
>>>>> "Tom" == Tom Lane <tgl@sss.pgh.pa.us> writes:
Tom> We can't inline wCTEs (those containing insert/update/delete)
Tom> without risk of semantics change.
Clearly.
Tom> I'd also not favor changing the semantics for CTEs that are read
Tom> more than once by the parent query.
This one's more debatable. There will still be cases where a CTE
referenced multiple times will be better inlined.
(It's obviously trivial to make the posted code do it that way, just by
checking cterefcount.)
Tom> However, a singly-referenced SELECT CTE could reasonably be
Tom> treated as equivalent to a sub-select-in-FROM,
In the PoC code I also excluded SELECT FOR UPDATE from inlining.
(There's already a difference between how SELECT FOR UPDATE works for
CTEs compared to subqueries and views, the comments mention it)
There might also be some merit in checking for volatility?
--
Andrew (irc:RhodiumToad)
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-24 23:03 ` Re: Early WIP/PoC for inlining CTEs Jeremy Finzel <finzelj@gmail.com>
2018-07-24 23:10 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
@ 2018-07-24 23:50 ` Gavin Flower <GavinFlower@archidevsys.co.nz>
3 siblings, 0 replies; 112+ messages in thread
From: Gavin Flower @ 2018-07-24 23:50 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; Jeremy Finzel <finzelj@gmail.com>; +Cc: Andrew Gierth <andrew@tao11.riddles.org.uk>; pgsql-hackers
On 25/07/18 11:10, Andres Freund wrote:
> On 2018-07-24 18:03:43 -0500, Jeremy Finzel wrote:
>> On Tue, Jul 24, 2018 at 5:28 PM Andrew Gierth <andrew@tao11.riddles.org.uk>
>> wrote:
>>
[...]
>>> In our environment we often want this to be a fence. For example it can
[...]
> This essentially has been discussed already:
> http://archives.postgresql.org/message-id/5351711493487900%40web53g.yandex.ru
>
> My read of the concensus (in which I am in the majority, so I might be
> biased) is that we do want inlining to be the default. We were thinking
> that it'd be necessary to provide a way to force inlining on the SQL
> level for individual CTEs.
>
>
>> Curious what other RDBMSs do here?
> They largely inline by default.
>
> Greetings,
>
> Andres Freund
>
If I'd not read anything about CTE's being a fence, I would have
implicitly assumed that they were optimised together with the main part
of the SQL statement, and I suspect that is the case for most people.
So I'm very much a favour of optimisation of CTE's being the default.
Cheers,
Gavin
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-24 23:03 ` Re: Early WIP/PoC for inlining CTEs Jeremy Finzel <finzelj@gmail.com>
2018-07-24 23:10 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
@ 2019-01-11 21:25 ` Merlin Moncure <mmoncure@gmail.com>
3 siblings, 0 replies; 112+ messages in thread
From: Merlin Moncure @ 2019-01-11 21:25 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Jeremy Finzel <finzelj@gmail.com>; Andrew Gierth <andrew@tao11.riddles.org.uk>; pgsql-hackers
On Tue, Jul 24, 2018 at 6:10 PM Andres Freund <andres@anarazel.de> wrote:
> My read of the concensus (in which I am in the majority, so I might be
> biased) is that we do want inlining to be the default. We were thinking
> that it'd be necessary to provide a way to force inlining on the SQL
> level for individual CTEs.
This is correct. Suggesting that we need syntax to disabling inlining
at the CTE level, and/or GUC to control the behavior (which I agree
should be defualted to inline). Something like
enable_cte_inline=true; I'm not very enthusiastic about explicitly
breaking intentionally introduced optimization fences and then forcing
people to inject our OFFSET 0 hack. This is just too unpleasant to
contemplate...what happens if we come up with a better implemntation
of OFFSET? yuck.
Thanks for providing this, CTE plan problems are a real bugaboo.
merlin
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
@ 2018-07-25 05:42 ` David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 15:32 ` Re: Early WIP/PoC for inlining CTEs Nico Williams <nico@cryptonector.com>
1 sibling, 2 replies; 112+ messages in thread
From: David Fetter @ 2018-07-25 05:42 UTC (permalink / raw)
To: Andrew Gierth <andrew@tao11.riddles.org.uk>; +Cc: pgsql-hackers
On Tue, Jul 24, 2018 at 11:28:21PM +0100, Andrew Gierth wrote:
> About a year ago I was briefly in discussion/collaboration with Adam Sah
> regarding the topic of inlining CTEs into the query rather than treating
> them as optimization barriers. We didn't take it very far (he sent me
> some stuff, I wrote some stuff and sent it back, things kind of got
> dropped at that point); but there's been some recent discussion of this
> and some people have expressed an interest in seeing the code.
>
> So I'm posting the parts that I wrote for the benefit of anyone wanting
> to pick up the issue again. The assumption of this code is that some
> form of syntax would exist to mark materialized CTEs and set the
> "ctematerialized" flag.
>
> I haven't rebased this or tested it since last year; this patch is
> against b81eba6a65.
Please find attached a version rebased atop 167075be3ab1547e18 with
what I believe are appropriate changes to regression test output. The
other changes to the regression tests output are somewhat puzzling, as
they change the actual results of queries. I've also attached both
the "leftover" diff and the files to which it should be applied.
Best,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778
Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate
*** /home/shackle/pggit/postgresql/src/test/regress/expected/window.out 2018-07-24 22:14:09.015838790 -0700
--- /home/shackle/pggit/postgresql/src/test/regress/results/window.out 2018-07-24 22:20:36.766079578 -0700
***************
*** 3297,3303 ****
);
eq1 | eq2 | eq3
-----+-----+-----
! t | t | t
(1 row)
--
--- 3297,3303 ----
);
eq1 | eq2 | eq3
-----+-----+-----
! f | f | t
(1 row)
--
======================================================================
*** /home/shackle/pggit/postgresql/src/test/regress/expected/with.out 2018-07-24 22:14:09.017838781 -0700
--- /home/shackle/pggit/postgresql/src/test/regress/results/with.out 2018-07-24 22:20:37.562075966 -0700
***************
*** 18,24 ****
) ss;
count
-------
! 5
(1 row)
-- WITH RECURSIVE
--- 18,24 ----
) ss;
count
-------
! 10
(1 row)
-- WITH RECURSIVE
======================================================================
--
-- Tests for common table expressions (WITH query, ... SELECT ...)
--
-- Basic WITH
WITH q1(x,y) AS (SELECT 1,2)
SELECT * FROM q1, q1 AS q2;
x | y | x | y
---+---+---+---
1 | 2 | 1 | 2
(1 row)
-- Multiple uses are evaluated only once
SELECT count(*) FROM (
WITH q1(x) AS (SELECT random() FROM generate_series(1, 5))
SELECT * FROM q1
UNION
SELECT * FROM q1
) ss;
count
-------
5
(1 row)
-- WITH RECURSIVE
-- sum of 1..100
WITH RECURSIVE t(n) AS (
VALUES (1)
UNION ALL
SELECT n+1 FROM t WHERE n < 100
)
SELECT sum(n) FROM t;
sum
------
5050
(1 row)
WITH RECURSIVE t(n) AS (
SELECT (VALUES(1))
UNION ALL
SELECT n+1 FROM t WHERE n < 5
)
SELECT * FROM t;
n
---
1
2
3
4
5
(5 rows)
-- recursive view
CREATE RECURSIVE VIEW nums (n) AS
VALUES (1)
UNION ALL
SELECT n+1 FROM nums WHERE n < 5;
SELECT * FROM nums;
n
---
1
2
3
4
5
(5 rows)
CREATE OR REPLACE RECURSIVE VIEW nums (n) AS
VALUES (1)
UNION ALL
SELECT n+1 FROM nums WHERE n < 6;
SELECT * FROM nums;
n
---
1
2
3
4
5
6
(6 rows)
-- This is an infinite loop with UNION ALL, but not with UNION
WITH RECURSIVE t(n) AS (
SELECT 1
UNION
SELECT 10-n FROM t)
SELECT * FROM t;
n
---
1
9
(2 rows)
-- This'd be an infinite loop, but outside query reads only as much as needed
WITH RECURSIVE t(n) AS (
VALUES (1)
UNION ALL
SELECT n+1 FROM t)
SELECT * FROM t LIMIT 10;
n
----
1
2
3
4
5
6
7
8
9
10
(10 rows)
-- UNION case should have same property
WITH RECURSIVE t(n) AS (
SELECT 1
UNION
SELECT n+1 FROM t)
SELECT * FROM t LIMIT 10;
n
----
1
2
3
4
5
6
7
8
9
10
(10 rows)
-- Test behavior with an unknown-type literal in the WITH
WITH q AS (SELECT 'foo' AS x)
SELECT x, x IS OF (text) AS is_text FROM q;
x | is_text
-----+---------
foo | t
(1 row)
WITH RECURSIVE t(n) AS (
SELECT 'foo'
UNION ALL
SELECT n || ' bar' FROM t WHERE length(n) < 20
)
SELECT n, n IS OF (text) AS is_text FROM t;
n | is_text
-------------------------+---------
foo | t
foo bar | t
foo bar bar | t
foo bar bar bar | t
foo bar bar bar bar | t
foo bar bar bar bar bar | t
(6 rows)
-- In a perfect world, this would work and resolve the literal as int ...
-- but for now, we have to be content with resolving to text too soon.
WITH RECURSIVE t(n) AS (
SELECT '7'
UNION ALL
SELECT n+1 FROM t WHERE n < 10
)
SELECT n, n IS OF (int) AS is_int FROM t;
ERROR: operator does not exist: text + integer
LINE 4: SELECT n+1 FROM t WHERE n < 10
^
HINT: No operator matches the given name and argument types. You might need to add explicit type casts.
--
-- Some examples with a tree
--
-- department structure represented here is as follows:
--
-- ROOT-+->A-+->B-+->C
-- | |
-- | +->D-+->F
-- +->E-+->G
CREATE TEMP TABLE department (
id INTEGER PRIMARY KEY, -- department ID
parent_department INTEGER REFERENCES department, -- upper department ID
name TEXT -- department name
);
INSERT INTO department VALUES (0, NULL, 'ROOT');
INSERT INTO department VALUES (1, 0, 'A');
INSERT INTO department VALUES (2, 1, 'B');
INSERT INTO department VALUES (3, 2, 'C');
INSERT INTO department VALUES (4, 2, 'D');
INSERT INTO department VALUES (5, 0, 'E');
INSERT INTO department VALUES (6, 4, 'F');
INSERT INTO department VALUES (7, 5, 'G');
-- extract all departments under 'A'. Result should be A, B, C, D and F
WITH RECURSIVE subdepartment AS
(
-- non recursive term
SELECT name as root_name, * FROM department WHERE name = 'A'
UNION ALL
-- recursive term
SELECT sd.root_name, d.* FROM department AS d, subdepartment AS sd
WHERE d.parent_department = sd.id
)
SELECT * FROM subdepartment ORDER BY name;
root_name | id | parent_department | name
-----------+----+-------------------+------
A | 1 | 0 | A
A | 2 | 1 | B
A | 3 | 2 | C
A | 4 | 2 | D
A | 6 | 4 | F
(5 rows)
-- extract all departments under 'A' with "level" number
WITH RECURSIVE subdepartment(level, id, parent_department, name) AS
(
-- non recursive term
SELECT 1, * FROM department WHERE name = 'A'
UNION ALL
-- recursive term
SELECT sd.level + 1, d.* FROM department AS d, subdepartment AS sd
WHERE d.parent_department = sd.id
)
SELECT * FROM subdepartment ORDER BY name;
level | id | parent_department | name
-------+----+-------------------+------
1 | 1 | 0 | A
2 | 2 | 1 | B
3 | 3 | 2 | C
3 | 4 | 2 | D
4 | 6 | 4 | F
(5 rows)
-- extract all departments under 'A' with "level" number.
-- Only shows level 2 or more
WITH RECURSIVE subdepartment(level, id, parent_department, name) AS
(
-- non recursive term
SELECT 1, * FROM department WHERE name = 'A'
UNION ALL
-- recursive term
SELECT sd.level + 1, d.* FROM department AS d, subdepartment AS sd
WHERE d.parent_department = sd.id
)
SELECT * FROM subdepartment WHERE level >= 2 ORDER BY name;
level | id | parent_department | name
-------+----+-------------------+------
2 | 2 | 1 | B
3 | 3 | 2 | C
3 | 4 | 2 | D
4 | 6 | 4 | F
(4 rows)
-- "RECURSIVE" is ignored if the query has no self-reference
WITH RECURSIVE subdepartment AS
(
-- note lack of recursive UNION structure
SELECT * FROM department WHERE name = 'A'
)
SELECT * FROM subdepartment ORDER BY name;
id | parent_department | name
----+-------------------+------
1 | 0 | A
(1 row)
-- inside subqueries
SELECT count(*) FROM (
WITH RECURSIVE t(n) AS (
SELECT 1 UNION ALL SELECT n + 1 FROM t WHERE n < 500
)
SELECT * FROM t) AS t WHERE n < (
SELECT count(*) FROM (
WITH RECURSIVE t(n) AS (
SELECT 1 UNION ALL SELECT n + 1 FROM t WHERE n < 100
)
SELECT * FROM t WHERE n < 50000
) AS t WHERE n < 100);
count
-------
98
(1 row)
-- use same CTE twice at different subquery levels
WITH q1(x,y) AS (
SELECT hundred, sum(ten) FROM tenk1 GROUP BY hundred
)
SELECT count(*) FROM q1 WHERE y > (SELECT sum(y)/100 FROM q1 qsub);
count
-------
50
(1 row)
-- via a VIEW
CREATE TEMPORARY VIEW vsubdepartment AS
WITH RECURSIVE subdepartment AS
(
-- non recursive term
SELECT * FROM department WHERE name = 'A'
UNION ALL
-- recursive term
SELECT d.* FROM department AS d, subdepartment AS sd
WHERE d.parent_department = sd.id
)
SELECT * FROM subdepartment;
SELECT * FROM vsubdepartment ORDER BY name;
id | parent_department | name
----+-------------------+------
1 | 0 | A
2 | 1 | B
3 | 2 | C
4 | 2 | D
6 | 4 | F
(5 rows)
-- Check reverse listing
SELECT pg_get_viewdef('vsubdepartment'::regclass);
pg_get_viewdef
-----------------------------------------------
WITH RECURSIVE subdepartment AS ( +
SELECT department.id, +
department.parent_department, +
department.name +
FROM department +
WHERE (department.name = 'A'::text)+
UNION ALL +
SELECT d.id, +
d.parent_department, +
d.name +
FROM department d, +
subdepartment sd +
WHERE (d.parent_department = sd.id)+
) +
SELECT subdepartment.id, +
subdepartment.parent_department, +
subdepartment.name +
FROM subdepartment;
(1 row)
SELECT pg_get_viewdef('vsubdepartment'::regclass, true);
pg_get_viewdef
---------------------------------------------
WITH RECURSIVE subdepartment AS ( +
SELECT department.id, +
department.parent_department, +
department.name +
FROM department +
WHERE department.name = 'A'::text+
UNION ALL +
SELECT d.id, +
d.parent_department, +
d.name +
FROM department d, +
subdepartment sd +
WHERE d.parent_department = sd.id+
) +
SELECT subdepartment.id, +
subdepartment.parent_department, +
subdepartment.name +
FROM subdepartment;
(1 row)
-- Another reverse-listing example
CREATE VIEW sums_1_100 AS
WITH RECURSIVE t(n) AS (
VALUES (1)
UNION ALL
SELECT n+1 FROM t WHERE n < 100
)
SELECT sum(n) FROM t;
\d+ sums_1_100
View "public.sums_1_100"
Column | Type | Collation | Nullable | Default | Storage | Description
--------+--------+-----------+----------+---------+---------+-------------
sum | bigint | | | | plain |
View definition:
WITH RECURSIVE t(n) AS (
VALUES (1)
UNION ALL
SELECT t_1.n + 1
FROM t t_1
WHERE t_1.n < 100
)
SELECT sum(t.n) AS sum
FROM t;
-- corner case in which sub-WITH gets initialized first
with recursive q as (
select * from department
union all
(with x as (select * from q)
select * from x)
)
select * from q limit 24;
id | parent_department | name
----+-------------------+------
0 | | ROOT
1 | 0 | A
2 | 1 | B
3 | 2 | C
4 | 2 | D
5 | 0 | E
6 | 4 | F
7 | 5 | G
0 | | ROOT
1 | 0 | A
2 | 1 | B
3 | 2 | C
4 | 2 | D
5 | 0 | E
6 | 4 | F
7 | 5 | G
0 | | ROOT
1 | 0 | A
2 | 1 | B
3 | 2 | C
4 | 2 | D
5 | 0 | E
6 | 4 | F
7 | 5 | G
(24 rows)
with recursive q as (
select * from department
union all
(with recursive x as (
select * from department
union all
(select * from q union all select * from x)
)
select * from x)
)
select * from q limit 32;
id | parent_department | name
----+-------------------+------
0 | | ROOT
1 | 0 | A
2 | 1 | B
3 | 2 | C
4 | 2 | D
5 | 0 | E
6 | 4 | F
7 | 5 | G
0 | | ROOT
1 | 0 | A
2 | 1 | B
3 | 2 | C
4 | 2 | D
5 | 0 | E
6 | 4 | F
7 | 5 | G
0 | | ROOT
1 | 0 | A
2 | 1 | B
3 | 2 | C
4 | 2 | D
5 | 0 | E
6 | 4 | F
7 | 5 | G
0 | | ROOT
1 | 0 | A
2 | 1 | B
3 | 2 | C
4 | 2 | D
5 | 0 | E
6 | 4 | F
7 | 5 | G
(32 rows)
-- recursive term has sub-UNION
WITH RECURSIVE t(i,j) AS (
VALUES (1,2)
UNION ALL
SELECT t2.i, t.j+1 FROM
(SELECT 2 AS i UNION ALL SELECT 3 AS i) AS t2
JOIN t ON (t2.i = t.i+1))
SELECT * FROM t;
i | j
---+---
1 | 2
2 | 3
3 | 4
(3 rows)
--
-- different tree example
--
CREATE TEMPORARY TABLE tree(
id INTEGER PRIMARY KEY,
parent_id INTEGER REFERENCES tree(id)
);
INSERT INTO tree
VALUES (1, NULL), (2, 1), (3,1), (4,2), (5,2), (6,2), (7,3), (8,3),
(9,4), (10,4), (11,7), (12,7), (13,7), (14, 9), (15,11), (16,11);
--
-- get all paths from "second level" nodes to leaf nodes
--
WITH RECURSIVE t(id, path) AS (
VALUES(1,ARRAY[]::integer[])
UNION ALL
SELECT tree.id, t.path || tree.id
FROM tree JOIN t ON (tree.parent_id = t.id)
)
SELECT t1.*, t2.* FROM t AS t1 JOIN t AS t2 ON
(t1.path[1] = t2.path[1] AND
array_upper(t1.path,1) = 1 AND
array_upper(t2.path,1) > 1)
ORDER BY t1.id, t2.id;
id | path | id | path
----+------+----+-------------
2 | {2} | 4 | {2,4}
2 | {2} | 5 | {2,5}
2 | {2} | 6 | {2,6}
2 | {2} | 9 | {2,4,9}
2 | {2} | 10 | {2,4,10}
2 | {2} | 14 | {2,4,9,14}
3 | {3} | 7 | {3,7}
3 | {3} | 8 | {3,8}
3 | {3} | 11 | {3,7,11}
3 | {3} | 12 | {3,7,12}
3 | {3} | 13 | {3,7,13}
3 | {3} | 15 | {3,7,11,15}
3 | {3} | 16 | {3,7,11,16}
(13 rows)
-- just count 'em
WITH RECURSIVE t(id, path) AS (
VALUES(1,ARRAY[]::integer[])
UNION ALL
SELECT tree.id, t.path || tree.id
FROM tree JOIN t ON (tree.parent_id = t.id)
)
SELECT t1.id, count(t2.*) FROM t AS t1 JOIN t AS t2 ON
(t1.path[1] = t2.path[1] AND
array_upper(t1.path,1) = 1 AND
array_upper(t2.path,1) > 1)
GROUP BY t1.id
ORDER BY t1.id;
id | count
----+-------
2 | 6
3 | 7
(2 rows)
-- this variant tickled a whole-row-variable bug in 8.4devel
WITH RECURSIVE t(id, path) AS (
VALUES(1,ARRAY[]::integer[])
UNION ALL
SELECT tree.id, t.path || tree.id
FROM tree JOIN t ON (tree.parent_id = t.id)
)
SELECT t1.id, t2.path, t2 FROM t AS t1 JOIN t AS t2 ON
(t1.id=t2.id);
id | path | t2
----+-------------+--------------------
1 | {} | (1,{})
2 | {2} | (2,{2})
3 | {3} | (3,{3})
4 | {2,4} | (4,"{2,4}")
5 | {2,5} | (5,"{2,5}")
6 | {2,6} | (6,"{2,6}")
7 | {3,7} | (7,"{3,7}")
8 | {3,8} | (8,"{3,8}")
9 | {2,4,9} | (9,"{2,4,9}")
10 | {2,4,10} | (10,"{2,4,10}")
11 | {3,7,11} | (11,"{3,7,11}")
12 | {3,7,12} | (12,"{3,7,12}")
13 | {3,7,13} | (13,"{3,7,13}")
14 | {2,4,9,14} | (14,"{2,4,9,14}")
15 | {3,7,11,15} | (15,"{3,7,11,15}")
16 | {3,7,11,16} | (16,"{3,7,11,16}")
(16 rows)
--
-- test cycle detection
--
create temp table graph( f int, t int, label text );
insert into graph values
(1, 2, 'arc 1 -> 2'),
(1, 3, 'arc 1 -> 3'),
(2, 3, 'arc 2 -> 3'),
(1, 4, 'arc 1 -> 4'),
(4, 5, 'arc 4 -> 5'),
(5, 1, 'arc 5 -> 1');
with recursive search_graph(f, t, label, path, cycle) as (
select *, array[row(g.f, g.t)], false from graph g
union all
select g.*, path || row(g.f, g.t), row(g.f, g.t) = any(path)
from graph g, search_graph sg
where g.f = sg.t and not cycle
)
select * from search_graph;
f | t | label | path | cycle
---+---+------------+-------------------------------------------+-------
1 | 2 | arc 1 -> 2 | {"(1,2)"} | f
1 | 3 | arc 1 -> 3 | {"(1,3)"} | f
2 | 3 | arc 2 -> 3 | {"(2,3)"} | f
1 | 4 | arc 1 -> 4 | {"(1,4)"} | f
4 | 5 | arc 4 -> 5 | {"(4,5)"} | f
5 | 1 | arc 5 -> 1 | {"(5,1)"} | f
1 | 2 | arc 1 -> 2 | {"(5,1)","(1,2)"} | f
1 | 3 | arc 1 -> 3 | {"(5,1)","(1,3)"} | f
1 | 4 | arc 1 -> 4 | {"(5,1)","(1,4)"} | f
2 | 3 | arc 2 -> 3 | {"(1,2)","(2,3)"} | f
4 | 5 | arc 4 -> 5 | {"(1,4)","(4,5)"} | f
5 | 1 | arc 5 -> 1 | {"(4,5)","(5,1)"} | f
1 | 2 | arc 1 -> 2 | {"(4,5)","(5,1)","(1,2)"} | f
1 | 3 | arc 1 -> 3 | {"(4,5)","(5,1)","(1,3)"} | f
1 | 4 | arc 1 -> 4 | {"(4,5)","(5,1)","(1,4)"} | f
2 | 3 | arc 2 -> 3 | {"(5,1)","(1,2)","(2,3)"} | f
4 | 5 | arc 4 -> 5 | {"(5,1)","(1,4)","(4,5)"} | f
5 | 1 | arc 5 -> 1 | {"(1,4)","(4,5)","(5,1)"} | f
1 | 2 | arc 1 -> 2 | {"(1,4)","(4,5)","(5,1)","(1,2)"} | f
1 | 3 | arc 1 -> 3 | {"(1,4)","(4,5)","(5,1)","(1,3)"} | f
1 | 4 | arc 1 -> 4 | {"(1,4)","(4,5)","(5,1)","(1,4)"} | t
2 | 3 | arc 2 -> 3 | {"(4,5)","(5,1)","(1,2)","(2,3)"} | f
4 | 5 | arc 4 -> 5 | {"(4,5)","(5,1)","(1,4)","(4,5)"} | t
5 | 1 | arc 5 -> 1 | {"(5,1)","(1,4)","(4,5)","(5,1)"} | t
2 | 3 | arc 2 -> 3 | {"(1,4)","(4,5)","(5,1)","(1,2)","(2,3)"} | f
(25 rows)
-- ordering by the path column has same effect as SEARCH DEPTH FIRST
with recursive search_graph(f, t, label, path, cycle) as (
select *, array[row(g.f, g.t)], false from graph g
union all
select g.*, path || row(g.f, g.t), row(g.f, g.t) = any(path)
from graph g, search_graph sg
where g.f = sg.t and not cycle
)
select * from search_graph order by path;
f | t | label | path | cycle
---+---+------------+-------------------------------------------+-------
1 | 2 | arc 1 -> 2 | {"(1,2)"} | f
2 | 3 | arc 2 -> 3 | {"(1,2)","(2,3)"} | f
1 | 3 | arc 1 -> 3 | {"(1,3)"} | f
1 | 4 | arc 1 -> 4 | {"(1,4)"} | f
4 | 5 | arc 4 -> 5 | {"(1,4)","(4,5)"} | f
5 | 1 | arc 5 -> 1 | {"(1,4)","(4,5)","(5,1)"} | f
1 | 2 | arc 1 -> 2 | {"(1,4)","(4,5)","(5,1)","(1,2)"} | f
2 | 3 | arc 2 -> 3 | {"(1,4)","(4,5)","(5,1)","(1,2)","(2,3)"} | f
1 | 3 | arc 1 -> 3 | {"(1,4)","(4,5)","(5,1)","(1,3)"} | f
1 | 4 | arc 1 -> 4 | {"(1,4)","(4,5)","(5,1)","(1,4)"} | t
2 | 3 | arc 2 -> 3 | {"(2,3)"} | f
4 | 5 | arc 4 -> 5 | {"(4,5)"} | f
5 | 1 | arc 5 -> 1 | {"(4,5)","(5,1)"} | f
1 | 2 | arc 1 -> 2 | {"(4,5)","(5,1)","(1,2)"} | f
2 | 3 | arc 2 -> 3 | {"(4,5)","(5,1)","(1,2)","(2,3)"} | f
1 | 3 | arc 1 -> 3 | {"(4,5)","(5,1)","(1,3)"} | f
1 | 4 | arc 1 -> 4 | {"(4,5)","(5,1)","(1,4)"} | f
4 | 5 | arc 4 -> 5 | {"(4,5)","(5,1)","(1,4)","(4,5)"} | t
5 | 1 | arc 5 -> 1 | {"(5,1)"} | f
1 | 2 | arc 1 -> 2 | {"(5,1)","(1,2)"} | f
2 | 3 | arc 2 -> 3 | {"(5,1)","(1,2)","(2,3)"} | f
1 | 3 | arc 1 -> 3 | {"(5,1)","(1,3)"} | f
1 | 4 | arc 1 -> 4 | {"(5,1)","(1,4)"} | f
4 | 5 | arc 4 -> 5 | {"(5,1)","(1,4)","(4,5)"} | f
5 | 1 | arc 5 -> 1 | {"(5,1)","(1,4)","(4,5)","(5,1)"} | t
(25 rows)
--
-- test multiple WITH queries
--
WITH RECURSIVE
y (id) AS (VALUES (1)),
x (id) AS (SELECT * FROM y UNION ALL SELECT id+1 FROM x WHERE id < 5)
SELECT * FROM x;
id
----
1
2
3
4
5
(5 rows)
-- forward reference OK
WITH RECURSIVE
x(id) AS (SELECT * FROM y UNION ALL SELECT id+1 FROM x WHERE id < 5),
y(id) AS (values (1))
SELECT * FROM x;
id
----
1
2
3
4
5
(5 rows)
WITH RECURSIVE
x(id) AS
(VALUES (1) UNION ALL SELECT id+1 FROM x WHERE id < 5),
y(id) AS
(VALUES (1) UNION ALL SELECT id+1 FROM y WHERE id < 10)
SELECT y.*, x.* FROM y LEFT JOIN x USING (id);
id | id
----+----
1 | 1
2 | 2
3 | 3
4 | 4
5 | 5
6 |
7 |
8 |
9 |
10 |
(10 rows)
WITH RECURSIVE
x(id) AS
(VALUES (1) UNION ALL SELECT id+1 FROM x WHERE id < 5),
y(id) AS
(VALUES (1) UNION ALL SELECT id+1 FROM x WHERE id < 10)
SELECT y.*, x.* FROM y LEFT JOIN x USING (id);
id | id
----+----
1 | 1
2 | 2
3 | 3
4 | 4
5 | 5
6 |
(6 rows)
WITH RECURSIVE
x(id) AS
(SELECT 1 UNION ALL SELECT id+1 FROM x WHERE id < 3 ),
y(id) AS
(SELECT * FROM x UNION ALL SELECT * FROM x),
z(id) AS
(SELECT * FROM x UNION ALL SELECT id+1 FROM z WHERE id < 10)
SELECT * FROM z;
id
----
1
2
3
2
3
4
3
4
5
4
5
6
5
6
7
6
7
8
7
8
9
8
9
10
9
10
10
(27 rows)
WITH RECURSIVE
x(id) AS
(SELECT 1 UNION ALL SELECT id+1 FROM x WHERE id < 3 ),
y(id) AS
(SELECT * FROM x UNION ALL SELECT * FROM x),
z(id) AS
(SELECT * FROM y UNION ALL SELECT id+1 FROM z WHERE id < 10)
SELECT * FROM z;
id
----
1
2
3
1
2
3
2
3
4
2
3
4
3
4
5
3
4
5
4
5
6
4
5
6
5
6
7
5
6
7
6
7
8
6
7
8
7
8
9
7
8
9
8
9
10
8
9
10
9
10
9
10
10
10
(54 rows)
--
-- Test WITH attached to a data-modifying statement
--
CREATE TEMPORARY TABLE y (a INTEGER);
INSERT INTO y SELECT generate_series(1, 10);
WITH t AS (
SELECT a FROM y
)
INSERT INTO y
SELECT a+20 FROM t RETURNING *;
a
----
21
22
23
24
25
26
27
28
29
30
(10 rows)
SELECT * FROM y;
a
----
1
2
3
4
5
6
7
8
9
10
21
22
23
24
25
26
27
28
29
30
(20 rows)
WITH t AS (
SELECT a FROM y
)
UPDATE y SET a = y.a-10 FROM t WHERE y.a > 20 AND t.a = y.a RETURNING y.a;
a
----
11
12
13
14
15
16
17
18
19
20
(10 rows)
SELECT * FROM y;
a
----
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
(20 rows)
WITH RECURSIVE t(a) AS (
SELECT 11
UNION ALL
SELECT a+1 FROM t WHERE a < 50
)
DELETE FROM y USING t WHERE t.a = y.a RETURNING y.a;
a
----
11
12
13
14
15
16
17
18
19
20
(10 rows)
SELECT * FROM y;
a
----
1
2
3
4
5
6
7
8
9
10
(10 rows)
DROP TABLE y;
--
-- error cases
--
-- INTERSECT
WITH RECURSIVE x(n) AS (SELECT 1 INTERSECT SELECT n+1 FROM x)
SELECT * FROM x;
ERROR: recursive query "x" does not have the form non-recursive-term UNION [ALL] recursive-term
LINE 1: WITH RECURSIVE x(n) AS (SELECT 1 INTERSECT SELECT n+1 FROM x...
^
WITH RECURSIVE x(n) AS (SELECT 1 INTERSECT ALL SELECT n+1 FROM x)
SELECT * FROM x;
ERROR: recursive query "x" does not have the form non-recursive-term UNION [ALL] recursive-term
LINE 1: WITH RECURSIVE x(n) AS (SELECT 1 INTERSECT ALL SELECT n+1 FR...
^
-- EXCEPT
WITH RECURSIVE x(n) AS (SELECT 1 EXCEPT SELECT n+1 FROM x)
SELECT * FROM x;
ERROR: recursive query "x" does not have the form non-recursive-term UNION [ALL] recursive-term
LINE 1: WITH RECURSIVE x(n) AS (SELECT 1 EXCEPT SELECT n+1 FROM x)
^
WITH RECURSIVE x(n) AS (SELECT 1 EXCEPT ALL SELECT n+1 FROM x)
SELECT * FROM x;
ERROR: recursive query "x" does not have the form non-recursive-term UNION [ALL] recursive-term
LINE 1: WITH RECURSIVE x(n) AS (SELECT 1 EXCEPT ALL SELECT n+1 FROM ...
^
-- no non-recursive term
WITH RECURSIVE x(n) AS (SELECT n FROM x)
SELECT * FROM x;
ERROR: recursive query "x" does not have the form non-recursive-term UNION [ALL] recursive-term
LINE 1: WITH RECURSIVE x(n) AS (SELECT n FROM x)
^
-- recursive term in the left hand side (strictly speaking, should allow this)
WITH RECURSIVE x(n) AS (SELECT n FROM x UNION ALL SELECT 1)
SELECT * FROM x;
ERROR: recursive reference to query "x" must not appear within its non-recursive term
LINE 1: WITH RECURSIVE x(n) AS (SELECT n FROM x UNION ALL SELECT 1)
^
CREATE TEMPORARY TABLE y (a INTEGER);
INSERT INTO y SELECT generate_series(1, 10);
-- LEFT JOIN
WITH RECURSIVE x(n) AS (SELECT a FROM y WHERE a = 1
UNION ALL
SELECT x.n+1 FROM y LEFT JOIN x ON x.n = y.a WHERE n < 10)
SELECT * FROM x;
ERROR: recursive reference to query "x" must not appear within an outer join
LINE 3: SELECT x.n+1 FROM y LEFT JOIN x ON x.n = y.a WHERE n < 10)
^
-- RIGHT JOIN
WITH RECURSIVE x(n) AS (SELECT a FROM y WHERE a = 1
UNION ALL
SELECT x.n+1 FROM x RIGHT JOIN y ON x.n = y.a WHERE n < 10)
SELECT * FROM x;
ERROR: recursive reference to query "x" must not appear within an outer join
LINE 3: SELECT x.n+1 FROM x RIGHT JOIN y ON x.n = y.a WHERE n < 10)
^
-- FULL JOIN
WITH RECURSIVE x(n) AS (SELECT a FROM y WHERE a = 1
UNION ALL
SELECT x.n+1 FROM x FULL JOIN y ON x.n = y.a WHERE n < 10)
SELECT * FROM x;
ERROR: recursive reference to query "x" must not appear within an outer join
LINE 3: SELECT x.n+1 FROM x FULL JOIN y ON x.n = y.a WHERE n < 10)
^
-- subquery
WITH RECURSIVE x(n) AS (SELECT 1 UNION ALL SELECT n+1 FROM x
WHERE n IN (SELECT * FROM x))
SELECT * FROM x;
ERROR: recursive reference to query "x" must not appear within a subquery
LINE 2: WHERE n IN (SELECT * FROM x))
^
-- aggregate functions
WITH RECURSIVE x(n) AS (SELECT 1 UNION ALL SELECT count(*) FROM x)
SELECT * FROM x;
ERROR: aggregate functions are not allowed in a recursive query's recursive term
LINE 1: WITH RECURSIVE x(n) AS (SELECT 1 UNION ALL SELECT count(*) F...
^
WITH RECURSIVE x(n) AS (SELECT 1 UNION ALL SELECT sum(n) FROM x)
SELECT * FROM x;
ERROR: aggregate functions are not allowed in a recursive query's recursive term
LINE 1: WITH RECURSIVE x(n) AS (SELECT 1 UNION ALL SELECT sum(n) FRO...
^
-- ORDER BY
WITH RECURSIVE x(n) AS (SELECT 1 UNION ALL SELECT n+1 FROM x ORDER BY 1)
SELECT * FROM x;
ERROR: ORDER BY in a recursive query is not implemented
LINE 1: ...VE x(n) AS (SELECT 1 UNION ALL SELECT n+1 FROM x ORDER BY 1)
^
-- LIMIT/OFFSET
WITH RECURSIVE x(n) AS (SELECT 1 UNION ALL SELECT n+1 FROM x LIMIT 10 OFFSET 1)
SELECT * FROM x;
ERROR: OFFSET in a recursive query is not implemented
LINE 1: ... AS (SELECT 1 UNION ALL SELECT n+1 FROM x LIMIT 10 OFFSET 1)
^
-- FOR UPDATE
WITH RECURSIVE x(n) AS (SELECT 1 UNION ALL SELECT n+1 FROM x FOR UPDATE)
SELECT * FROM x;
ERROR: FOR UPDATE/SHARE in a recursive query is not implemented
-- target list has a recursive query name
WITH RECURSIVE x(id) AS (values (1)
UNION ALL
SELECT (SELECT * FROM x) FROM x WHERE id < 5
) SELECT * FROM x;
ERROR: recursive reference to query "x" must not appear within a subquery
LINE 3: SELECT (SELECT * FROM x) FROM x WHERE id < 5
^
-- mutual recursive query (not implemented)
WITH RECURSIVE
x (id) AS (SELECT 1 UNION ALL SELECT id+1 FROM y WHERE id < 5),
y (id) AS (SELECT 1 UNION ALL SELECT id+1 FROM x WHERE id < 5)
SELECT * FROM x;
ERROR: mutual recursion between WITH items is not implemented
LINE 2: x (id) AS (SELECT 1 UNION ALL SELECT id+1 FROM y WHERE id ...
^
-- non-linear recursion is not allowed
WITH RECURSIVE foo(i) AS
(values (1)
UNION ALL
(SELECT i+1 FROM foo WHERE i < 10
UNION ALL
SELECT i+1 FROM foo WHERE i < 5)
) SELECT * FROM foo;
ERROR: recursive reference to query "foo" must not appear more than once
LINE 6: SELECT i+1 FROM foo WHERE i < 5)
^
WITH RECURSIVE foo(i) AS
(values (1)
UNION ALL
SELECT * FROM
(SELECT i+1 FROM foo WHERE i < 10
UNION ALL
SELECT i+1 FROM foo WHERE i < 5) AS t
) SELECT * FROM foo;
ERROR: recursive reference to query "foo" must not appear more than once
LINE 7: SELECT i+1 FROM foo WHERE i < 5) AS t
^
WITH RECURSIVE foo(i) AS
(values (1)
UNION ALL
(SELECT i+1 FROM foo WHERE i < 10
EXCEPT
SELECT i+1 FROM foo WHERE i < 5)
) SELECT * FROM foo;
ERROR: recursive reference to query "foo" must not appear within EXCEPT
LINE 6: SELECT i+1 FROM foo WHERE i < 5)
^
WITH RECURSIVE foo(i) AS
(values (1)
UNION ALL
(SELECT i+1 FROM foo WHERE i < 10
INTERSECT
SELECT i+1 FROM foo WHERE i < 5)
) SELECT * FROM foo;
ERROR: recursive reference to query "foo" must not appear more than once
LINE 6: SELECT i+1 FROM foo WHERE i < 5)
^
-- Wrong type induced from non-recursive term
WITH RECURSIVE foo(i) AS
(SELECT i FROM (VALUES(1),(2)) t(i)
UNION ALL
SELECT (i+1)::numeric(10,0) FROM foo WHERE i < 10)
SELECT * FROM foo;
ERROR: recursive query "foo" column 1 has type integer in non-recursive term but type numeric overall
LINE 2: (SELECT i FROM (VALUES(1),(2)) t(i)
^
HINT: Cast the output of the non-recursive term to the correct type.
-- rejects different typmod, too (should we allow this?)
WITH RECURSIVE foo(i) AS
(SELECT i::numeric(3,0) FROM (VALUES(1),(2)) t(i)
UNION ALL
SELECT (i+1)::numeric(10,0) FROM foo WHERE i < 10)
SELECT * FROM foo;
ERROR: recursive query "foo" column 1 has type numeric(3,0) in non-recursive term but type numeric overall
LINE 2: (SELECT i::numeric(3,0) FROM (VALUES(1),(2)) t(i)
^
HINT: Cast the output of the non-recursive term to the correct type.
-- disallow OLD/NEW reference in CTE
CREATE TEMPORARY TABLE x (n integer);
CREATE RULE r2 AS ON UPDATE TO x DO INSTEAD
WITH t AS (SELECT OLD.*) UPDATE y SET a = t.n FROM t;
ERROR: cannot refer to OLD within WITH query
--
-- test for bug #4902
--
with cte(foo) as ( values(42) ) values((select foo from cte));
column1
---------
42
(1 row)
with cte(foo) as ( select 42 ) select * from ((select foo from cte)) q;
foo
-----
42
(1 row)
-- test CTE referencing an outer-level variable (to see that changed-parameter
-- signaling still works properly after fixing this bug)
select ( with cte(foo) as ( values(f1) )
select (select foo from cte) )
from int4_tbl;
foo
-------------
0
123456
-123456
2147483647
-2147483647
(5 rows)
select ( with cte(foo) as ( values(f1) )
values((select foo from cte)) )
from int4_tbl;
column1
-------------
0
123456
-123456
2147483647
-2147483647
(5 rows)
--
-- test for nested-recursive-WITH bug
--
WITH RECURSIVE t(j) AS (
WITH RECURSIVE s(i) AS (
VALUES (1)
UNION ALL
SELECT i+1 FROM s WHERE i < 10
)
SELECT i FROM s
UNION ALL
SELECT j+1 FROM t WHERE j < 10
)
SELECT * FROM t;
j
----
1
2
3
4
5
6
7
8
9
10
2
3
4
5
6
7
8
9
10
3
4
5
6
7
8
9
10
4
5
6
7
8
9
10
5
6
7
8
9
10
6
7
8
9
10
7
8
9
10
8
9
10
9
10
10
(55 rows)
--
-- test WITH attached to intermediate-level set operation
--
WITH outermost(x) AS (
SELECT 1
UNION (WITH innermost as (SELECT 2)
SELECT * FROM innermost
UNION SELECT 3)
)
SELECT * FROM outermost ORDER BY 1;
x
---
1
2
3
(3 rows)
WITH outermost(x) AS (
SELECT 1
UNION (WITH innermost as (SELECT 2)
SELECT * FROM outermost -- fail
UNION SELECT * FROM innermost)
)
SELECT * FROM outermost ORDER BY 1;
ERROR: relation "outermost" does not exist
LINE 4: SELECT * FROM outermost
^
DETAIL: There is a WITH item named "outermost", but it cannot be referenced from this part of the query.
HINT: Use WITH RECURSIVE, or re-order the WITH items to remove forward references.
WITH RECURSIVE outermost(x) AS (
SELECT 1
UNION (WITH innermost as (SELECT 2)
SELECT * FROM outermost
UNION SELECT * FROM innermost)
)
SELECT * FROM outermost ORDER BY 1;
x
---
1
2
(2 rows)
WITH RECURSIVE outermost(x) AS (
WITH innermost as (SELECT 2 FROM outermost) -- fail
SELECT * FROM innermost
UNION SELECT * from outermost
)
SELECT * FROM outermost ORDER BY 1;
ERROR: recursive reference to query "outermost" must not appear within a subquery
LINE 2: WITH innermost as (SELECT 2 FROM outermost)
^
--
-- This test will fail with the old implementation of PARAM_EXEC parameter
-- assignment, because the "q1" Var passed down to A's targetlist subselect
-- looks exactly like the "A.id" Var passed down to C's subselect, causing
-- the old code to give them the same runtime PARAM_EXEC slot. But the
-- lifespans of the two parameters overlap, thanks to B also reading A.
--
with
A as ( select q2 as id, (select q1) as x from int8_tbl ),
B as ( select id, row_number() over (partition by id) as r from A ),
C as ( select A.id, array(select B.id from B where B.id = A.id) from A )
select * from C;
id | array
-------------------+-------------------------------------
456 | {456}
4567890123456789 | {4567890123456789,4567890123456789}
123 | {123}
4567890123456789 | {4567890123456789,4567890123456789}
-4567890123456789 | {-4567890123456789}
(5 rows)
--
-- Test CTEs read in non-initialization orders
--
WITH RECURSIVE
tab(id_key,link) AS (VALUES (1,17), (2,17), (3,17), (4,17), (6,17), (5,17)),
iter (id_key, row_type, link) AS (
SELECT 0, 'base', 17
UNION ALL (
WITH remaining(id_key, row_type, link, min) AS (
SELECT tab.id_key, 'true'::text, iter.link, MIN(tab.id_key) OVER ()
FROM tab INNER JOIN iter USING (link)
WHERE tab.id_key > iter.id_key
),
first_remaining AS (
SELECT id_key, row_type, link
FROM remaining
WHERE id_key=min
),
effect AS (
SELECT tab.id_key, 'new'::text, tab.link
FROM first_remaining e INNER JOIN tab ON e.id_key=tab.id_key
WHERE e.row_type = 'false'
)
SELECT * FROM first_remaining
UNION ALL SELECT * FROM effect
)
)
SELECT * FROM iter;
id_key | row_type | link
--------+----------+------
0 | base | 17
1 | true | 17
2 | true | 17
3 | true | 17
4 | true | 17
5 | true | 17
6 | true | 17
(7 rows)
WITH RECURSIVE
tab(id_key,link) AS (VALUES (1,17), (2,17), (3,17), (4,17), (6,17), (5,17)),
iter (id_key, row_type, link) AS (
SELECT 0, 'base', 17
UNION (
WITH remaining(id_key, row_type, link, min) AS (
SELECT tab.id_key, 'true'::text, iter.link, MIN(tab.id_key) OVER ()
FROM tab INNER JOIN iter USING (link)
WHERE tab.id_key > iter.id_key
),
first_remaining AS (
SELECT id_key, row_type, link
FROM remaining
WHERE id_key=min
),
effect AS (
SELECT tab.id_key, 'new'::text, tab.link
FROM first_remaining e INNER JOIN tab ON e.id_key=tab.id_key
WHERE e.row_type = 'false'
)
SELECT * FROM first_remaining
UNION ALL SELECT * FROM effect
)
)
SELECT * FROM iter;
id_key | row_type | link
--------+----------+------
0 | base | 17
1 | true | 17
2 | true | 17
3 | true | 17
4 | true | 17
5 | true | 17
6 | true | 17
(7 rows)
--
-- Data-modifying statements in WITH
--
-- INSERT ... RETURNING
WITH t AS (
INSERT INTO y
VALUES
(11),
(12),
(13),
(14),
(15),
(16),
(17),
(18),
(19),
(20)
RETURNING *
)
SELECT * FROM t;
a
----
11
12
13
14
15
16
17
18
19
20
(10 rows)
SELECT * FROM y;
a
----
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
(20 rows)
-- UPDATE ... RETURNING
WITH t AS (
UPDATE y
SET a=a+1
RETURNING *
)
SELECT * FROM t;
a
----
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
(20 rows)
SELECT * FROM y;
a
----
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
(20 rows)
-- DELETE ... RETURNING
WITH t AS (
DELETE FROM y
WHERE a <= 10
RETURNING *
)
SELECT * FROM t;
a
----
2
3
4
5
6
7
8
9
10
(9 rows)
SELECT * FROM y;
a
----
11
12
13
14
15
16
17
18
19
20
21
(11 rows)
-- forward reference
WITH RECURSIVE t AS (
INSERT INTO y
SELECT a+5 FROM t2 WHERE a > 5
RETURNING *
), t2 AS (
UPDATE y SET a=a-11 RETURNING *
)
SELECT * FROM t
UNION ALL
SELECT * FROM t2;
a
----
11
12
13
14
15
0
1
2
3
4
5
6
7
8
9
10
(16 rows)
SELECT * FROM y;
a
----
0
1
2
3
4
5
6
11
7
12
8
13
9
14
10
15
(16 rows)
-- unconditional DO INSTEAD rule
CREATE RULE y_rule AS ON DELETE TO y DO INSTEAD
INSERT INTO y VALUES(42) RETURNING *;
WITH t AS (
DELETE FROM y RETURNING *
)
SELECT * FROM t;
a
----
42
(1 row)
SELECT * FROM y;
a
----
0
1
2
3
4
5
6
11
7
12
8
13
9
14
10
15
42
(17 rows)
DROP RULE y_rule ON y;
-- check merging of outer CTE with CTE in a rule action
CREATE TEMP TABLE bug6051 AS
select i from generate_series(1,3) as t(i);
SELECT * FROM bug6051;
i
---
1
2
3
(3 rows)
WITH t1 AS ( DELETE FROM bug6051 RETURNING * )
INSERT INTO bug6051 SELECT * FROM t1;
SELECT * FROM bug6051;
i
---
1
2
3
(3 rows)
CREATE TEMP TABLE bug6051_2 (i int);
CREATE RULE bug6051_ins AS ON INSERT TO bug6051 DO INSTEAD
INSERT INTO bug6051_2
SELECT NEW.i;
WITH t1 AS ( DELETE FROM bug6051 RETURNING * )
INSERT INTO bug6051 SELECT * FROM t1;
SELECT * FROM bug6051;
i
---
(0 rows)
SELECT * FROM bug6051_2;
i
---
1
2
3
(3 rows)
-- a truly recursive CTE in the same list
WITH RECURSIVE t(a) AS (
SELECT 0
UNION ALL
SELECT a+1 FROM t WHERE a+1 < 5
), t2 as (
INSERT INTO y
SELECT * FROM t RETURNING *
)
SELECT * FROM t2 JOIN y USING (a) ORDER BY a;
a
---
0
1
2
3
4
(5 rows)
SELECT * FROM y;
a
----
0
1
2
3
4
5
6
11
7
12
8
13
9
14
10
15
42
0
1
2
3
4
(22 rows)
-- data-modifying WITH in a modifying statement
WITH t AS (
DELETE FROM y
WHERE a <= 10
RETURNING *
)
INSERT INTO y SELECT -a FROM t RETURNING *;
a
-----
0
-1
-2
-3
-4
-5
-6
-7
-8
-9
-10
0
-1
-2
-3
-4
(16 rows)
SELECT * FROM y;
a
-----
11
12
13
14
15
42
0
-1
-2
-3
-4
-5
-6
-7
-8
-9
-10
0
-1
-2
-3
-4
(22 rows)
-- check that WITH query is run to completion even if outer query isn't
WITH t AS (
UPDATE y SET a = a * 100 RETURNING *
)
SELECT * FROM t LIMIT 10;
a
------
1100
1200
1300
1400
1500
4200
0
-100
-200
-300
(10 rows)
SELECT * FROM y;
a
-------
1100
1200
1300
1400
1500
4200
0
-100
-200
-300
-400
-500
-600
-700
-800
-900
-1000
0
-100
-200
-300
-400
(22 rows)
-- data-modifying WITH containing INSERT...ON CONFLICT DO UPDATE
CREATE TABLE withz AS SELECT i AS k, (i || ' v')::text v FROM generate_series(1, 16, 3) i;
ALTER TABLE withz ADD UNIQUE (k);
WITH t AS (
INSERT INTO withz SELECT i, 'insert'
FROM generate_series(0, 16) i
ON CONFLICT (k) DO UPDATE SET v = withz.v || ', now update'
RETURNING *
)
SELECT * FROM t JOIN y ON t.k = y.a ORDER BY a, k;
k | v | a
---+--------+---
0 | insert | 0
0 | insert | 0
(2 rows)
-- Test EXCLUDED.* reference within CTE
WITH aa AS (
INSERT INTO withz VALUES(1, 5) ON CONFLICT (k) DO UPDATE SET v = EXCLUDED.v
WHERE withz.k != EXCLUDED.k
RETURNING *
)
SELECT * FROM aa;
k | v
---+---
(0 rows)
-- New query/snapshot demonstrates side-effects of previous query.
SELECT * FROM withz ORDER BY k;
k | v
----+------------------
0 | insert
1 | 1 v, now update
2 | insert
3 | insert
4 | 4 v, now update
5 | insert
6 | insert
7 | 7 v, now update
8 | insert
9 | insert
10 | 10 v, now update
11 | insert
12 | insert
13 | 13 v, now update
14 | insert
15 | insert
16 | 16 v, now update
(17 rows)
--
-- Ensure subqueries within the update clause work, even if they
-- reference outside values
--
WITH aa AS (SELECT 1 a, 2 b)
INSERT INTO withz VALUES(1, 'insert')
ON CONFLICT (k) DO UPDATE SET v = (SELECT b || ' update' FROM aa WHERE a = 1 LIMIT 1);
WITH aa AS (SELECT 1 a, 2 b)
INSERT INTO withz VALUES(1, 'insert')
ON CONFLICT (k) DO UPDATE SET v = ' update' WHERE withz.k = (SELECT a FROM aa);
WITH aa AS (SELECT 1 a, 2 b)
INSERT INTO withz VALUES(1, 'insert')
ON CONFLICT (k) DO UPDATE SET v = (SELECT b || ' update' FROM aa WHERE a = 1 LIMIT 1);
WITH aa AS (SELECT 'a' a, 'b' b UNION ALL SELECT 'a' a, 'b' b)
INSERT INTO withz VALUES(1, 'insert')
ON CONFLICT (k) DO UPDATE SET v = (SELECT b || ' update' FROM aa WHERE a = 'a' LIMIT 1);
WITH aa AS (SELECT 1 a, 2 b)
INSERT INTO withz VALUES(1, (SELECT b || ' insert' FROM aa WHERE a = 1 ))
ON CONFLICT (k) DO UPDATE SET v = (SELECT b || ' update' FROM aa WHERE a = 1 LIMIT 1);
-- Update a row more than once, in different parts of a wCTE. That is
-- an allowed, presumably very rare, edge case, but since it was
-- broken in the past, having a test seems worthwhile.
WITH simpletup AS (
SELECT 2 k, 'Green' v),
upsert_cte AS (
INSERT INTO withz VALUES(2, 'Blue') ON CONFLICT (k) DO
UPDATE SET (k, v) = (SELECT k, v FROM simpletup WHERE simpletup.k = withz.k)
RETURNING k, v)
INSERT INTO withz VALUES(2, 'Red') ON CONFLICT (k) DO
UPDATE SET (k, v) = (SELECT k, v FROM upsert_cte WHERE upsert_cte.k = withz.k)
RETURNING k, v;
k | v
---+---
(0 rows)
DROP TABLE withz;
-- check that run to completion happens in proper ordering
TRUNCATE TABLE y;
INSERT INTO y SELECT generate_series(1, 3);
CREATE TEMPORARY TABLE yy (a INTEGER);
WITH RECURSIVE t1 AS (
INSERT INTO y SELECT * FROM y RETURNING *
), t2 AS (
INSERT INTO yy SELECT * FROM t1 RETURNING *
)
SELECT 1;
?column?
----------
1
(1 row)
SELECT * FROM y;
a
---
1
2
3
1
2
3
(6 rows)
SELECT * FROM yy;
a
---
1
2
3
(3 rows)
WITH RECURSIVE t1 AS (
INSERT INTO yy SELECT * FROM t2 RETURNING *
), t2 AS (
INSERT INTO y SELECT * FROM y RETURNING *
)
SELECT 1;
?column?
----------
1
(1 row)
SELECT * FROM y;
a
---
1
2
3
1
2
3
1
2
3
1
2
3
(12 rows)
SELECT * FROM yy;
a
---
1
2
3
1
2
3
1
2
3
(9 rows)
-- triggers
TRUNCATE TABLE y;
INSERT INTO y SELECT generate_series(1, 10);
CREATE FUNCTION y_trigger() RETURNS trigger AS $$
begin
raise notice 'y_trigger: a = %', new.a;
return new;
end;
$$ LANGUAGE plpgsql;
CREATE TRIGGER y_trig BEFORE INSERT ON y FOR EACH ROW
EXECUTE PROCEDURE y_trigger();
WITH t AS (
INSERT INTO y
VALUES
(21),
(22),
(23)
RETURNING *
)
SELECT * FROM t;
NOTICE: y_trigger: a = 21
NOTICE: y_trigger: a = 22
NOTICE: y_trigger: a = 23
a
----
21
22
23
(3 rows)
SELECT * FROM y;
a
----
1
2
3
4
5
6
7
8
9
10
21
22
23
(13 rows)
DROP TRIGGER y_trig ON y;
CREATE TRIGGER y_trig AFTER INSERT ON y FOR EACH ROW
EXECUTE PROCEDURE y_trigger();
WITH t AS (
INSERT INTO y
VALUES
(31),
(32),
(33)
RETURNING *
)
SELECT * FROM t LIMIT 1;
NOTICE: y_trigger: a = 31
NOTICE: y_trigger: a = 32
NOTICE: y_trigger: a = 33
a
----
31
(1 row)
SELECT * FROM y;
a
----
1
2
3
4
5
6
7
8
9
10
21
22
23
31
32
33
(16 rows)
DROP TRIGGER y_trig ON y;
CREATE OR REPLACE FUNCTION y_trigger() RETURNS trigger AS $$
begin
raise notice 'y_trigger';
return null;
end;
$$ LANGUAGE plpgsql;
CREATE TRIGGER y_trig AFTER INSERT ON y FOR EACH STATEMENT
EXECUTE PROCEDURE y_trigger();
WITH t AS (
INSERT INTO y
VALUES
(41),
(42),
(43)
RETURNING *
)
SELECT * FROM t;
NOTICE: y_trigger
a
----
41
42
43
(3 rows)
SELECT * FROM y;
a
----
1
2
3
4
5
6
7
8
9
10
21
22
23
31
32
33
41
42
43
(19 rows)
DROP TRIGGER y_trig ON y;
DROP FUNCTION y_trigger();
-- WITH attached to inherited UPDATE or DELETE
CREATE TEMP TABLE parent ( id int, val text );
CREATE TEMP TABLE child1 ( ) INHERITS ( parent );
CREATE TEMP TABLE child2 ( ) INHERITS ( parent );
INSERT INTO parent VALUES ( 1, 'p1' );
INSERT INTO child1 VALUES ( 11, 'c11' ),( 12, 'c12' );
INSERT INTO child2 VALUES ( 23, 'c21' ),( 24, 'c22' );
WITH rcte AS ( SELECT sum(id) AS totalid FROM parent )
UPDATE parent SET id = id + totalid FROM rcte;
SELECT * FROM parent;
id | val
----+-----
72 | p1
82 | c11
83 | c12
94 | c21
95 | c22
(5 rows)
WITH wcte AS ( INSERT INTO child1 VALUES ( 42, 'new' ) RETURNING id AS newid )
UPDATE parent SET id = id + newid FROM wcte;
SELECT * FROM parent;
id | val
-----+-----
114 | p1
42 | new
124 | c11
125 | c12
136 | c21
137 | c22
(6 rows)
WITH rcte AS ( SELECT max(id) AS maxid FROM parent )
DELETE FROM parent USING rcte WHERE id = maxid;
SELECT * FROM parent;
id | val
-----+-----
114 | p1
42 | new
124 | c11
125 | c12
136 | c21
(5 rows)
WITH wcte AS ( INSERT INTO child2 VALUES ( 42, 'new2' ) RETURNING id AS newid )
DELETE FROM parent USING wcte WHERE id = newid;
SELECT * FROM parent;
id | val
-----+------
114 | p1
124 | c11
125 | c12
136 | c21
42 | new2
(5 rows)
-- check EXPLAIN VERBOSE for a wCTE with RETURNING
EXPLAIN (VERBOSE, COSTS OFF)
WITH wcte AS ( INSERT INTO int8_tbl VALUES ( 42, 47 ) RETURNING q2 )
DELETE FROM a USING wcte WHERE aa = q2;
QUERY PLAN
----------------------------------------------------
Delete on public.a
Delete on public.a
Delete on public.b
Delete on public.c
Delete on public.d
CTE wcte
-> Insert on public.int8_tbl
Output: int8_tbl.q2
-> Result
Output: '42'::bigint, '47'::bigint
-> Nested Loop
Output: a.ctid, wcte.*
Join Filter: (a.aa = wcte.q2)
-> Seq Scan on public.a
Output: a.ctid, a.aa
-> CTE Scan on wcte
Output: wcte.*, wcte.q2
-> Nested Loop
Output: b.ctid, wcte.*
Join Filter: (b.aa = wcte.q2)
-> Seq Scan on public.b
Output: b.ctid, b.aa
-> CTE Scan on wcte
Output: wcte.*, wcte.q2
-> Nested Loop
Output: c.ctid, wcte.*
Join Filter: (c.aa = wcte.q2)
-> Seq Scan on public.c
Output: c.ctid, c.aa
-> CTE Scan on wcte
Output: wcte.*, wcte.q2
-> Nested Loop
Output: d.ctid, wcte.*
Join Filter: (d.aa = wcte.q2)
-> Seq Scan on public.d
Output: d.ctid, d.aa
-> CTE Scan on wcte
Output: wcte.*, wcte.q2
(38 rows)
-- error cases
-- data-modifying WITH tries to use its own output
WITH RECURSIVE t AS (
INSERT INTO y
SELECT * FROM t
)
VALUES(FALSE);
ERROR: recursive query "t" must not contain data-modifying statements
LINE 1: WITH RECURSIVE t AS (
^
-- no RETURNING in a referenced data-modifying WITH
WITH t AS (
INSERT INTO y VALUES(0)
)
SELECT * FROM t;
ERROR: WITH query "t" does not have a RETURNING clause
LINE 4: SELECT * FROM t;
^
-- data-modifying WITH allowed only at the top level
SELECT * FROM (
WITH t AS (UPDATE y SET a=a+1 RETURNING *)
SELECT * FROM t
) ss;
ERROR: WITH clause containing a data-modifying statement must be at the top level
LINE 2: WITH t AS (UPDATE y SET a=a+1 RETURNING *)
^
-- most variants of rules aren't allowed
CREATE RULE y_rule AS ON INSERT TO y WHERE a=0 DO INSTEAD DELETE FROM y;
WITH t AS (
INSERT INTO y VALUES(0)
)
VALUES(FALSE);
ERROR: conditional DO INSTEAD rules are not supported for data-modifying statements in WITH
DROP RULE y_rule ON y;
-- check that parser lookahead for WITH doesn't cause any odd behavior
create table foo (with baz); -- fail, WITH is a reserved word
ERROR: syntax error at or near "with"
LINE 1: create table foo (with baz);
^
create table foo (with ordinality); -- fail, WITH is a reserved word
ERROR: syntax error at or near "with"
LINE 1: create table foo (with ordinality);
^
with ordinality as (select 1 as x) select * from ordinality;
x
---
1
(1 row)
-- check sane response to attempt to modify CTE relation
WITH test AS (SELECT 42) INSERT INTO test VALUES (1);
ERROR: relation "test" does not exist
LINE 1: WITH test AS (SELECT 42) INSERT INTO test VALUES (1);
^
-- check response to attempt to modify table with same name as a CTE (perhaps
-- surprisingly it works, because CTEs don't hide tables from data-modifying
-- statements)
create temp table test (i int);
with test as (select 42) insert into test select * from test;
select * from test;
i
----
42
(1 row)
drop table test;
--
-- WINDOW FUNCTIONS
--
CREATE TEMPORARY TABLE empsalary (
depname varchar,
empno bigint,
salary int,
enroll_date date
);
INSERT INTO empsalary VALUES
('develop', 10, 5200, '2007-08-01'),
('sales', 1, 5000, '2006-10-01'),
('personnel', 5, 3500, '2007-12-10'),
('sales', 4, 4800, '2007-08-08'),
('personnel', 2, 3900, '2006-12-23'),
('develop', 7, 4200, '2008-01-01'),
('develop', 9, 4500, '2008-01-01'),
('sales', 3, 4800, '2007-08-01'),
('develop', 8, 6000, '2006-10-01'),
('develop', 11, 5200, '2007-08-15');
SELECT depname, empno, salary, sum(salary) OVER (PARTITION BY depname) FROM empsalary ORDER BY depname, salary;
depname | empno | salary | sum
-----------+-------+--------+-------
develop | 7 | 4200 | 25100
develop | 9 | 4500 | 25100
develop | 11 | 5200 | 25100
develop | 10 | 5200 | 25100
develop | 8 | 6000 | 25100
personnel | 5 | 3500 | 7400
personnel | 2 | 3900 | 7400
sales | 3 | 4800 | 14600
sales | 4 | 4800 | 14600
sales | 1 | 5000 | 14600
(10 rows)
SELECT depname, empno, salary, rank() OVER (PARTITION BY depname ORDER BY salary) FROM empsalary;
depname | empno | salary | rank
-----------+-------+--------+------
develop | 7 | 4200 | 1
develop | 9 | 4500 | 2
develop | 11 | 5200 | 3
develop | 10 | 5200 | 3
develop | 8 | 6000 | 5
personnel | 5 | 3500 | 1
personnel | 2 | 3900 | 2
sales | 3 | 4800 | 1
sales | 4 | 4800 | 1
sales | 1 | 5000 | 3
(10 rows)
-- with GROUP BY
SELECT four, ten, SUM(SUM(four)) OVER (PARTITION BY four), AVG(ten) FROM tenk1
GROUP BY four, ten ORDER BY four, ten;
four | ten | sum | avg
------+-----+------+------------------------
0 | 0 | 0 | 0.00000000000000000000
0 | 2 | 0 | 2.0000000000000000
0 | 4 | 0 | 4.0000000000000000
0 | 6 | 0 | 6.0000000000000000
0 | 8 | 0 | 8.0000000000000000
1 | 1 | 2500 | 1.00000000000000000000
1 | 3 | 2500 | 3.0000000000000000
1 | 5 | 2500 | 5.0000000000000000
1 | 7 | 2500 | 7.0000000000000000
1 | 9 | 2500 | 9.0000000000000000
2 | 0 | 5000 | 0.00000000000000000000
2 | 2 | 5000 | 2.0000000000000000
2 | 4 | 5000 | 4.0000000000000000
2 | 6 | 5000 | 6.0000000000000000
2 | 8 | 5000 | 8.0000000000000000
3 | 1 | 7500 | 1.00000000000000000000
3 | 3 | 7500 | 3.0000000000000000
3 | 5 | 7500 | 5.0000000000000000
3 | 7 | 7500 | 7.0000000000000000
3 | 9 | 7500 | 9.0000000000000000
(20 rows)
SELECT depname, empno, salary, sum(salary) OVER w FROM empsalary WINDOW w AS (PARTITION BY depname);
depname | empno | salary | sum
-----------+-------+--------+-------
develop | 11 | 5200 | 25100
develop | 7 | 4200 | 25100
develop | 9 | 4500 | 25100
develop | 8 | 6000 | 25100
develop | 10 | 5200 | 25100
personnel | 5 | 3500 | 7400
personnel | 2 | 3900 | 7400
sales | 3 | 4800 | 14600
sales | 1 | 5000 | 14600
sales | 4 | 4800 | 14600
(10 rows)
SELECT depname, empno, salary, rank() OVER w FROM empsalary WINDOW w AS (PARTITION BY depname ORDER BY salary) ORDER BY rank() OVER w;
depname | empno | salary | rank
-----------+-------+--------+------
develop | 7 | 4200 | 1
personnel | 5 | 3500 | 1
sales | 3 | 4800 | 1
sales | 4 | 4800 | 1
personnel | 2 | 3900 | 2
develop | 9 | 4500 | 2
sales | 1 | 5000 | 3
develop | 11 | 5200 | 3
develop | 10 | 5200 | 3
develop | 8 | 6000 | 5
(10 rows)
-- empty window specification
SELECT COUNT(*) OVER () FROM tenk1 WHERE unique2 < 10;
count
-------
10
10
10
10
10
10
10
10
10
10
(10 rows)
SELECT COUNT(*) OVER w FROM tenk1 WHERE unique2 < 10 WINDOW w AS ();
count
-------
10
10
10
10
10
10
10
10
10
10
(10 rows)
-- no window operation
SELECT four FROM tenk1 WHERE FALSE WINDOW w AS (PARTITION BY ten);
four
------
(0 rows)
-- cumulative aggregate
SELECT sum(four) OVER (PARTITION BY ten ORDER BY unique2) AS sum_1, ten, four FROM tenk1 WHERE unique2 < 10;
sum_1 | ten | four
-------+-----+------
0 | 0 | 0
0 | 0 | 0
2 | 0 | 2
3 | 1 | 3
4 | 1 | 1
5 | 1 | 1
3 | 3 | 3
0 | 4 | 0
1 | 7 | 1
1 | 9 | 1
(10 rows)
SELECT row_number() OVER (ORDER BY unique2) FROM tenk1 WHERE unique2 < 10;
row_number
------------
1
2
3
4
5
6
7
8
9
10
(10 rows)
SELECT rank() OVER (PARTITION BY four ORDER BY ten) AS rank_1, ten, four FROM tenk1 WHERE unique2 < 10;
rank_1 | ten | four
--------+-----+------
1 | 0 | 0
1 | 0 | 0
3 | 4 | 0
1 | 1 | 1
1 | 1 | 1
3 | 7 | 1
4 | 9 | 1
1 | 0 | 2
1 | 1 | 3
2 | 3 | 3
(10 rows)
SELECT dense_rank() OVER (PARTITION BY four ORDER BY ten), ten, four FROM tenk1 WHERE unique2 < 10;
dense_rank | ten | four
------------+-----+------
1 | 0 | 0
1 | 0 | 0
2 | 4 | 0
1 | 1 | 1
1 | 1 | 1
2 | 7 | 1
3 | 9 | 1
1 | 0 | 2
1 | 1 | 3
2 | 3 | 3
(10 rows)
SELECT percent_rank() OVER (PARTITION BY four ORDER BY ten), ten, four FROM tenk1 WHERE unique2 < 10;
percent_rank | ten | four
-------------------+-----+------
0 | 0 | 0
0 | 0 | 0
1 | 4 | 0
0 | 1 | 1
0 | 1 | 1
0.666666666666667 | 7 | 1
1 | 9 | 1
0 | 0 | 2
0 | 1 | 3
1 | 3 | 3
(10 rows)
SELECT cume_dist() OVER (PARTITION BY four ORDER BY ten), ten, four FROM tenk1 WHERE unique2 < 10;
cume_dist | ten | four
-------------------+-----+------
0.666666666666667 | 0 | 0
0.666666666666667 | 0 | 0
1 | 4 | 0
0.5 | 1 | 1
0.5 | 1 | 1
0.75 | 7 | 1
1 | 9 | 1
1 | 0 | 2
0.5 | 1 | 3
1 | 3 | 3
(10 rows)
SELECT ntile(3) OVER (ORDER BY ten, four), ten, four FROM tenk1 WHERE unique2 < 10;
ntile | ten | four
-------+-----+------
1 | 0 | 0
1 | 0 | 0
1 | 0 | 2
1 | 1 | 1
2 | 1 | 1
2 | 1 | 3
2 | 3 | 3
3 | 4 | 0
3 | 7 | 1
3 | 9 | 1
(10 rows)
SELECT ntile(NULL) OVER (ORDER BY ten, four), ten, four FROM tenk1 LIMIT 2;
ntile | ten | four
-------+-----+------
| 0 | 0
| 0 | 0
(2 rows)
SELECT lag(ten) OVER (PARTITION BY four ORDER BY ten), ten, four FROM tenk1 WHERE unique2 < 10;
lag | ten | four
-----+-----+------
| 0 | 0
0 | 0 | 0
0 | 4 | 0
| 1 | 1
1 | 1 | 1
1 | 7 | 1
7 | 9 | 1
| 0 | 2
| 1 | 3
1 | 3 | 3
(10 rows)
SELECT lag(ten, four) OVER (PARTITION BY four ORDER BY ten), ten, four FROM tenk1 WHERE unique2 < 10;
lag | ten | four
-----+-----+------
0 | 0 | 0
0 | 0 | 0
4 | 4 | 0
| 1 | 1
1 | 1 | 1
1 | 7 | 1
7 | 9 | 1
| 0 | 2
| 1 | 3
| 3 | 3
(10 rows)
SELECT lag(ten, four, 0) OVER (PARTITION BY four ORDER BY ten), ten, four FROM tenk1 WHERE unique2 < 10;
lag | ten | four
-----+-----+------
0 | 0 | 0
0 | 0 | 0
4 | 4 | 0
0 | 1 | 1
1 | 1 | 1
1 | 7 | 1
7 | 9 | 1
0 | 0 | 2
0 | 1 | 3
0 | 3 | 3
(10 rows)
SELECT lead(ten) OVER (PARTITION BY four ORDER BY ten), ten, four FROM tenk1 WHERE unique2 < 10;
lead | ten | four
------+-----+------
0 | 0 | 0
4 | 0 | 0
| 4 | 0
1 | 1 | 1
7 | 1 | 1
9 | 7 | 1
| 9 | 1
| 0 | 2
3 | 1 | 3
| 3 | 3
(10 rows)
SELECT lead(ten * 2, 1) OVER (PARTITION BY four ORDER BY ten), ten, four FROM tenk1 WHERE unique2 < 10;
lead | ten | four
------+-----+------
0 | 0 | 0
8 | 0 | 0
| 4 | 0
2 | 1 | 1
14 | 1 | 1
18 | 7 | 1
| 9 | 1
| 0 | 2
6 | 1 | 3
| 3 | 3
(10 rows)
SELECT lead(ten * 2, 1, -1) OVER (PARTITION BY four ORDER BY ten), ten, four FROM tenk1 WHERE unique2 < 10;
lead | ten | four
------+-----+------
0 | 0 | 0
8 | 0 | 0
-1 | 4 | 0
2 | 1 | 1
14 | 1 | 1
18 | 7 | 1
-1 | 9 | 1
-1 | 0 | 2
6 | 1 | 3
-1 | 3 | 3
(10 rows)
SELECT first_value(ten) OVER (PARTITION BY four ORDER BY ten), ten, four FROM tenk1 WHERE unique2 < 10;
first_value | ten | four
-------------+-----+------
0 | 0 | 0
0 | 0 | 0
0 | 4 | 0
1 | 1 | 1
1 | 1 | 1
1 | 7 | 1
1 | 9 | 1
0 | 0 | 2
1 | 1 | 3
1 | 3 | 3
(10 rows)
-- last_value returns the last row of the frame, which is CURRENT ROW in ORDER BY window.
SELECT last_value(four) OVER (ORDER BY ten), ten, four FROM tenk1 WHERE unique2 < 10;
last_value | ten | four
------------+-----+------
0 | 0 | 0
0 | 0 | 2
0 | 0 | 0
1 | 1 | 1
1 | 1 | 3
1 | 1 | 1
3 | 3 | 3
0 | 4 | 0
1 | 7 | 1
1 | 9 | 1
(10 rows)
SELECT last_value(ten) OVER (PARTITION BY four), ten, four FROM
(SELECT * FROM tenk1 WHERE unique2 < 10 ORDER BY four, ten)s
ORDER BY four, ten;
last_value | ten | four
------------+-----+------
4 | 0 | 0
4 | 0 | 0
4 | 4 | 0
9 | 1 | 1
9 | 1 | 1
9 | 7 | 1
9 | 9 | 1
0 | 0 | 2
3 | 1 | 3
3 | 3 | 3
(10 rows)
SELECT nth_value(ten, four + 1) OVER (PARTITION BY four), ten, four
FROM (SELECT * FROM tenk1 WHERE unique2 < 10 ORDER BY four, ten)s;
nth_value | ten | four
-----------+-----+------
0 | 0 | 0
0 | 0 | 0
0 | 4 | 0
1 | 1 | 1
1 | 1 | 1
1 | 7 | 1
1 | 9 | 1
| 0 | 2
| 1 | 3
| 3 | 3
(10 rows)
SELECT ten, two, sum(hundred) AS gsum, sum(sum(hundred)) OVER (PARTITION BY two ORDER BY ten) AS wsum
FROM tenk1 GROUP BY ten, two;
ten | two | gsum | wsum
-----+-----+-------+--------
0 | 0 | 45000 | 45000
2 | 0 | 47000 | 92000
4 | 0 | 49000 | 141000
6 | 0 | 51000 | 192000
8 | 0 | 53000 | 245000
1 | 1 | 46000 | 46000
3 | 1 | 48000 | 94000
5 | 1 | 50000 | 144000
7 | 1 | 52000 | 196000
9 | 1 | 54000 | 250000
(10 rows)
SELECT count(*) OVER (PARTITION BY four), four FROM (SELECT * FROM tenk1 WHERE two = 1)s WHERE unique2 < 10;
count | four
-------+------
4 | 1
4 | 1
4 | 1
4 | 1
2 | 3
2 | 3
(6 rows)
SELECT (count(*) OVER (PARTITION BY four ORDER BY ten) +
sum(hundred) OVER (PARTITION BY four ORDER BY ten))::varchar AS cntsum
FROM tenk1 WHERE unique2 < 10;
cntsum
--------
22
22
87
24
24
82
92
51
92
136
(10 rows)
-- opexpr with different windows evaluation.
SELECT * FROM(
SELECT count(*) OVER (PARTITION BY four ORDER BY ten) +
sum(hundred) OVER (PARTITION BY two ORDER BY ten) AS total,
count(*) OVER (PARTITION BY four ORDER BY ten) AS fourcount,
sum(hundred) OVER (PARTITION BY two ORDER BY ten) AS twosum
FROM tenk1
)sub
WHERE total <> fourcount + twosum;
total | fourcount | twosum
-------+-----------+--------
(0 rows)
SELECT avg(four) OVER (PARTITION BY four ORDER BY thousand / 100) FROM tenk1 WHERE unique2 < 10;
avg
------------------------
0.00000000000000000000
0.00000000000000000000
0.00000000000000000000
1.00000000000000000000
1.00000000000000000000
1.00000000000000000000
1.00000000000000000000
2.0000000000000000
3.0000000000000000
3.0000000000000000
(10 rows)
SELECT ten, two, sum(hundred) AS gsum, sum(sum(hundred)) OVER win AS wsum
FROM tenk1 GROUP BY ten, two WINDOW win AS (PARTITION BY two ORDER BY ten);
ten | two | gsum | wsum
-----+-----+-------+--------
0 | 0 | 45000 | 45000
2 | 0 | 47000 | 92000
4 | 0 | 49000 | 141000
6 | 0 | 51000 | 192000
8 | 0 | 53000 | 245000
1 | 1 | 46000 | 46000
3 | 1 | 48000 | 94000
5 | 1 | 50000 | 144000
7 | 1 | 52000 | 196000
9 | 1 | 54000 | 250000
(10 rows)
-- more than one window with GROUP BY
SELECT sum(salary),
row_number() OVER (ORDER BY depname),
sum(sum(salary)) OVER (ORDER BY depname DESC)
FROM empsalary GROUP BY depname;
sum | row_number | sum
-------+------------+-------
14600 | 3 | 14600
7400 | 2 | 22000
25100 | 1 | 47100
(3 rows)
-- identical windows with different names
SELECT sum(salary) OVER w1, count(*) OVER w2
FROM empsalary WINDOW w1 AS (ORDER BY salary), w2 AS (ORDER BY salary);
sum | count
-------+-------
3500 | 1
7400 | 2
11600 | 3
16100 | 4
25700 | 6
25700 | 6
30700 | 7
41100 | 9
41100 | 9
47100 | 10
(10 rows)
-- subplan
SELECT lead(ten, (SELECT two FROM tenk1 WHERE s.unique2 = unique2)) OVER (PARTITION BY four ORDER BY ten)
FROM tenk1 s WHERE unique2 < 10;
lead
------
0
0
4
1
7
9
0
3
(10 rows)
-- empty table
SELECT count(*) OVER (PARTITION BY four) FROM (SELECT * FROM tenk1 WHERE FALSE)s;
count
-------
(0 rows)
-- mixture of agg/wfunc in the same window
SELECT sum(salary) OVER w, rank() OVER w FROM empsalary WINDOW w AS (PARTITION BY depname ORDER BY salary DESC);
sum | rank
-------+------
6000 | 1
16400 | 2
16400 | 2
20900 | 4
25100 | 5
3900 | 1
7400 | 2
5000 | 1
14600 | 2
14600 | 2
(10 rows)
-- strict aggs
SELECT empno, depname, salary, bonus, depadj, MIN(bonus) OVER (ORDER BY empno), MAX(depadj) OVER () FROM(
SELECT *,
CASE WHEN enroll_date < '2008-01-01' THEN 2008 - extract(YEAR FROM enroll_date) END * 500 AS bonus,
CASE WHEN
AVG(salary) OVER (PARTITION BY depname) < salary
THEN 200 END AS depadj FROM empsalary
)s;
empno | depname | salary | bonus | depadj | min | max
-------+-----------+--------+-------+--------+------+-----
1 | sales | 5000 | 1000 | 200 | 1000 | 200
2 | personnel | 3900 | 1000 | 200 | 1000 | 200
3 | sales | 4800 | 500 | | 500 | 200
4 | sales | 4800 | 500 | | 500 | 200
5 | personnel | 3500 | 500 | | 500 | 200
7 | develop | 4200 | | | 500 | 200
8 | develop | 6000 | 1000 | 200 | 500 | 200
9 | develop | 4500 | | | 500 | 200
10 | develop | 5200 | 500 | 200 | 500 | 200
11 | develop | 5200 | 500 | 200 | 500 | 200
(10 rows)
-- window function over ungrouped agg over empty row set (bug before 9.1)
SELECT SUM(COUNT(f1)) OVER () FROM int4_tbl WHERE f1=42;
sum
-----
0
(1 row)
-- window function with ORDER BY an expression involving aggregates (9.1 bug)
select ten,
sum(unique1) + sum(unique2) as res,
rank() over (order by sum(unique1) + sum(unique2)) as rank
from tenk1
group by ten order by ten;
ten | res | rank
-----+----------+------
0 | 9976146 | 4
1 | 10114187 | 9
2 | 10059554 | 8
3 | 9878541 | 1
4 | 9881005 | 2
5 | 9981670 | 5
6 | 9947099 | 3
7 | 10120309 | 10
8 | 9991305 | 6
9 | 10040184 | 7
(10 rows)
-- window and aggregate with GROUP BY expression (9.2 bug)
explain (costs off)
select first_value(max(x)) over (), y
from (select unique1 as x, ten+four as y from tenk1) ss
group by y;
QUERY PLAN
---------------------------------------------
WindowAgg
-> HashAggregate
Group Key: (tenk1.ten + tenk1.four)
-> Seq Scan on tenk1
(4 rows)
-- test non-default frame specifications
SELECT four, ten,
sum(ten) over (partition by four order by ten),
last_value(ten) over (partition by four order by ten)
FROM (select distinct ten, four from tenk1) ss;
four | ten | sum | last_value
------+-----+-----+------------
0 | 0 | 0 | 0
0 | 2 | 2 | 2
0 | 4 | 6 | 4
0 | 6 | 12 | 6
0 | 8 | 20 | 8
1 | 1 | 1 | 1
1 | 3 | 4 | 3
1 | 5 | 9 | 5
1 | 7 | 16 | 7
1 | 9 | 25 | 9
2 | 0 | 0 | 0
2 | 2 | 2 | 2
2 | 4 | 6 | 4
2 | 6 | 12 | 6
2 | 8 | 20 | 8
3 | 1 | 1 | 1
3 | 3 | 4 | 3
3 | 5 | 9 | 5
3 | 7 | 16 | 7
3 | 9 | 25 | 9
(20 rows)
SELECT four, ten,
sum(ten) over (partition by four order by ten range between unbounded preceding and current row),
last_value(ten) over (partition by four order by ten range between unbounded preceding and current row)
FROM (select distinct ten, four from tenk1) ss;
four | ten | sum | last_value
------+-----+-----+------------
0 | 0 | 0 | 0
0 | 2 | 2 | 2
0 | 4 | 6 | 4
0 | 6 | 12 | 6
0 | 8 | 20 | 8
1 | 1 | 1 | 1
1 | 3 | 4 | 3
1 | 5 | 9 | 5
1 | 7 | 16 | 7
1 | 9 | 25 | 9
2 | 0 | 0 | 0
2 | 2 | 2 | 2
2 | 4 | 6 | 4
2 | 6 | 12 | 6
2 | 8 | 20 | 8
3 | 1 | 1 | 1
3 | 3 | 4 | 3
3 | 5 | 9 | 5
3 | 7 | 16 | 7
3 | 9 | 25 | 9
(20 rows)
SELECT four, ten,
sum(ten) over (partition by four order by ten range between unbounded preceding and unbounded following),
last_value(ten) over (partition by four order by ten range between unbounded preceding and unbounded following)
FROM (select distinct ten, four from tenk1) ss;
four | ten | sum | last_value
------+-----+-----+------------
0 | 0 | 20 | 8
0 | 2 | 20 | 8
0 | 4 | 20 | 8
0 | 6 | 20 | 8
0 | 8 | 20 | 8
1 | 1 | 25 | 9
1 | 3 | 25 | 9
1 | 5 | 25 | 9
1 | 7 | 25 | 9
1 | 9 | 25 | 9
2 | 0 | 20 | 8
2 | 2 | 20 | 8
2 | 4 | 20 | 8
2 | 6 | 20 | 8
2 | 8 | 20 | 8
3 | 1 | 25 | 9
3 | 3 | 25 | 9
3 | 5 | 25 | 9
3 | 7 | 25 | 9
3 | 9 | 25 | 9
(20 rows)
SELECT four, ten/4 as two,
sum(ten/4) over (partition by four order by ten/4 range between unbounded preceding and current row),
last_value(ten/4) over (partition by four order by ten/4 range between unbounded preceding and current row)
FROM (select distinct ten, four from tenk1) ss;
four | two | sum | last_value
------+-----+-----+------------
0 | 0 | 0 | 0
0 | 0 | 0 | 0
0 | 1 | 2 | 1
0 | 1 | 2 | 1
0 | 2 | 4 | 2
1 | 0 | 0 | 0
1 | 0 | 0 | 0
1 | 1 | 2 | 1
1 | 1 | 2 | 1
1 | 2 | 4 | 2
2 | 0 | 0 | 0
2 | 0 | 0 | 0
2 | 1 | 2 | 1
2 | 1 | 2 | 1
2 | 2 | 4 | 2
3 | 0 | 0 | 0
3 | 0 | 0 | 0
3 | 1 | 2 | 1
3 | 1 | 2 | 1
3 | 2 | 4 | 2
(20 rows)
SELECT four, ten/4 as two,
sum(ten/4) over (partition by four order by ten/4 rows between unbounded preceding and current row),
last_value(ten/4) over (partition by four order by ten/4 rows between unbounded preceding and current row)
FROM (select distinct ten, four from tenk1) ss;
four | two | sum | last_value
------+-----+-----+------------
0 | 0 | 0 | 0
0 | 0 | 0 | 0
0 | 1 | 1 | 1
0 | 1 | 2 | 1
0 | 2 | 4 | 2
1 | 0 | 0 | 0
1 | 0 | 0 | 0
1 | 1 | 1 | 1
1 | 1 | 2 | 1
1 | 2 | 4 | 2
2 | 0 | 0 | 0
2 | 0 | 0 | 0
2 | 1 | 1 | 1
2 | 1 | 2 | 1
2 | 2 | 4 | 2
3 | 0 | 0 | 0
3 | 0 | 0 | 0
3 | 1 | 1 | 1
3 | 1 | 2 | 1
3 | 2 | 4 | 2
(20 rows)
SELECT sum(unique1) over (order by four range between current row and unbounded following),
unique1, four
FROM tenk1 WHERE unique1 < 10;
sum | unique1 | four
-----+---------+------
45 | 0 | 0
45 | 8 | 0
45 | 4 | 0
33 | 5 | 1
33 | 9 | 1
33 | 1 | 1
18 | 6 | 2
18 | 2 | 2
10 | 3 | 3
10 | 7 | 3
(10 rows)
SELECT sum(unique1) over (rows between current row and unbounded following),
unique1, four
FROM tenk1 WHERE unique1 < 10;
sum | unique1 | four
-----+---------+------
45 | 4 | 0
41 | 2 | 2
39 | 1 | 1
38 | 6 | 2
32 | 9 | 1
23 | 8 | 0
15 | 5 | 1
10 | 3 | 3
7 | 7 | 3
0 | 0 | 0
(10 rows)
SELECT sum(unique1) over (rows between 2 preceding and 2 following),
unique1, four
FROM tenk1 WHERE unique1 < 10;
sum | unique1 | four
-----+---------+------
7 | 4 | 0
13 | 2 | 2
22 | 1 | 1
26 | 6 | 2
29 | 9 | 1
31 | 8 | 0
32 | 5 | 1
23 | 3 | 3
15 | 7 | 3
10 | 0 | 0
(10 rows)
SELECT sum(unique1) over (rows between 2 preceding and 2 following exclude no others),
unique1, four
FROM tenk1 WHERE unique1 < 10;
sum | unique1 | four
-----+---------+------
7 | 4 | 0
13 | 2 | 2
22 | 1 | 1
26 | 6 | 2
29 | 9 | 1
31 | 8 | 0
32 | 5 | 1
23 | 3 | 3
15 | 7 | 3
10 | 0 | 0
(10 rows)
SELECT sum(unique1) over (rows between 2 preceding and 2 following exclude current row),
unique1, four
FROM tenk1 WHERE unique1 < 10;
sum | unique1 | four
-----+---------+------
3 | 4 | 0
11 | 2 | 2
21 | 1 | 1
20 | 6 | 2
20 | 9 | 1
23 | 8 | 0
27 | 5 | 1
20 | 3 | 3
8 | 7 | 3
10 | 0 | 0
(10 rows)
SELECT sum(unique1) over (rows between 2 preceding and 2 following exclude group),
unique1, four
FROM tenk1 WHERE unique1 < 10;
sum | unique1 | four
-----+---------+------
| 4 | 0
| 2 | 2
| 1 | 1
| 6 | 2
| 9 | 1
| 8 | 0
| 5 | 1
| 3 | 3
| 7 | 3
| 0 | 0
(10 rows)
SELECT sum(unique1) over (rows between 2 preceding and 2 following exclude ties),
unique1, four
FROM tenk1 WHERE unique1 < 10;
sum | unique1 | four
-----+---------+------
4 | 4 | 0
2 | 2 | 2
1 | 1 | 1
6 | 6 | 2
9 | 9 | 1
8 | 8 | 0
5 | 5 | 1
3 | 3 | 3
7 | 7 | 3
0 | 0 | 0
(10 rows)
SELECT first_value(unique1) over (ORDER BY four rows between current row and 2 following exclude current row),
unique1, four
FROM tenk1 WHERE unique1 < 10;
first_value | unique1 | four
-------------+---------+------
8 | 0 | 0
4 | 8 | 0
5 | 4 | 0
9 | 5 | 1
1 | 9 | 1
6 | 1 | 1
2 | 6 | 2
3 | 2 | 2
7 | 3 | 3
| 7 | 3
(10 rows)
SELECT first_value(unique1) over (ORDER BY four rows between current row and 2 following exclude group),
unique1, four
FROM tenk1 WHERE unique1 < 10;
first_value | unique1 | four
-------------+---------+------
| 0 | 0
5 | 8 | 0
5 | 4 | 0
| 5 | 1
6 | 9 | 1
6 | 1 | 1
3 | 6 | 2
3 | 2 | 2
| 3 | 3
| 7 | 3
(10 rows)
SELECT first_value(unique1) over (ORDER BY four rows between current row and 2 following exclude ties),
unique1, four
FROM tenk1 WHERE unique1 < 10;
first_value | unique1 | four
-------------+---------+------
0 | 0 | 0
8 | 8 | 0
4 | 4 | 0
5 | 5 | 1
9 | 9 | 1
1 | 1 | 1
6 | 6 | 2
2 | 2 | 2
3 | 3 | 3
7 | 7 | 3
(10 rows)
SELECT last_value(unique1) over (ORDER BY four rows between current row and 2 following exclude current row),
unique1, four
FROM tenk1 WHERE unique1 < 10;
last_value | unique1 | four
------------+---------+------
4 | 0 | 0
5 | 8 | 0
9 | 4 | 0
1 | 5 | 1
6 | 9 | 1
2 | 1 | 1
3 | 6 | 2
7 | 2 | 2
7 | 3 | 3
| 7 | 3
(10 rows)
SELECT last_value(unique1) over (ORDER BY four rows between current row and 2 following exclude group),
unique1, four
FROM tenk1 WHERE unique1 < 10;
last_value | unique1 | four
------------+---------+------
| 0 | 0
5 | 8 | 0
9 | 4 | 0
| 5 | 1
6 | 9 | 1
2 | 1 | 1
3 | 6 | 2
7 | 2 | 2
| 3 | 3
| 7 | 3
(10 rows)
SELECT last_value(unique1) over (ORDER BY four rows between current row and 2 following exclude ties),
unique1, four
FROM tenk1 WHERE unique1 < 10;
last_value | unique1 | four
------------+---------+------
0 | 0 | 0
5 | 8 | 0
9 | 4 | 0
5 | 5 | 1
6 | 9 | 1
2 | 1 | 1
3 | 6 | 2
7 | 2 | 2
3 | 3 | 3
7 | 7 | 3
(10 rows)
SELECT sum(unique1) over (rows between 2 preceding and 1 preceding),
unique1, four
FROM tenk1 WHERE unique1 < 10;
sum | unique1 | four
-----+---------+------
| 4 | 0
4 | 2 | 2
6 | 1 | 1
3 | 6 | 2
7 | 9 | 1
15 | 8 | 0
17 | 5 | 1
13 | 3 | 3
8 | 7 | 3
10 | 0 | 0
(10 rows)
SELECT sum(unique1) over (rows between 1 following and 3 following),
unique1, four
FROM tenk1 WHERE unique1 < 10;
sum | unique1 | four
-----+---------+------
9 | 4 | 0
16 | 2 | 2
23 | 1 | 1
22 | 6 | 2
16 | 9 | 1
15 | 8 | 0
10 | 5 | 1
7 | 3 | 3
0 | 7 | 3
| 0 | 0
(10 rows)
SELECT sum(unique1) over (rows between unbounded preceding and 1 following),
unique1, four
FROM tenk1 WHERE unique1 < 10;
sum | unique1 | four
-----+---------+------
6 | 4 | 0
7 | 2 | 2
13 | 1 | 1
22 | 6 | 2
30 | 9 | 1
35 | 8 | 0
38 | 5 | 1
45 | 3 | 3
45 | 7 | 3
45 | 0 | 0
(10 rows)
SELECT sum(unique1) over (w range between current row and unbounded following),
unique1, four
FROM tenk1 WHERE unique1 < 10 WINDOW w AS (order by four);
sum | unique1 | four
-----+---------+------
45 | 0 | 0
45 | 8 | 0
45 | 4 | 0
33 | 5 | 1
33 | 9 | 1
33 | 1 | 1
18 | 6 | 2
18 | 2 | 2
10 | 3 | 3
10 | 7 | 3
(10 rows)
SELECT sum(unique1) over (w range between unbounded preceding and current row exclude current row),
unique1, four
FROM tenk1 WHERE unique1 < 10 WINDOW w AS (order by four);
sum | unique1 | four
-----+---------+------
12 | 0 | 0
4 | 8 | 0
8 | 4 | 0
22 | 5 | 1
18 | 9 | 1
26 | 1 | 1
29 | 6 | 2
33 | 2 | 2
42 | 3 | 3
38 | 7 | 3
(10 rows)
SELECT sum(unique1) over (w range between unbounded preceding and current row exclude group),
unique1, four
FROM tenk1 WHERE unique1 < 10 WINDOW w AS (order by four);
sum | unique1 | four
-----+---------+------
| 0 | 0
| 8 | 0
| 4 | 0
12 | 5 | 1
12 | 9 | 1
12 | 1 | 1
27 | 6 | 2
27 | 2 | 2
35 | 3 | 3
35 | 7 | 3
(10 rows)
SELECT sum(unique1) over (w range between unbounded preceding and current row exclude ties),
unique1, four
FROM tenk1 WHERE unique1 < 10 WINDOW w AS (order by four);
sum | unique1 | four
-----+---------+------
0 | 0 | 0
8 | 8 | 0
4 | 4 | 0
17 | 5 | 1
21 | 9 | 1
13 | 1 | 1
33 | 6 | 2
29 | 2 | 2
38 | 3 | 3
42 | 7 | 3
(10 rows)
SELECT first_value(unique1) over w,
nth_value(unique1, 2) over w AS nth_2,
last_value(unique1) over w, unique1, four
FROM tenk1 WHERE unique1 < 10
WINDOW w AS (order by four range between current row and unbounded following);
first_value | nth_2 | last_value | unique1 | four
-------------+-------+------------+---------+------
0 | 8 | 7 | 0 | 0
0 | 8 | 7 | 8 | 0
0 | 8 | 7 | 4 | 0
5 | 9 | 7 | 5 | 1
5 | 9 | 7 | 9 | 1
5 | 9 | 7 | 1 | 1
6 | 2 | 7 | 6 | 2
6 | 2 | 7 | 2 | 2
3 | 7 | 7 | 3 | 3
3 | 7 | 7 | 7 | 3
(10 rows)
SELECT sum(unique1) over
(order by unique1
rows (SELECT unique1 FROM tenk1 ORDER BY unique1 LIMIT 1) + 1 PRECEDING),
unique1
FROM tenk1 WHERE unique1 < 10;
sum | unique1
-----+---------
0 | 0
1 | 1
3 | 2
5 | 3
7 | 4
9 | 5
11 | 6
13 | 7
15 | 8
17 | 9
(10 rows)
CREATE TEMP VIEW v_window AS
SELECT i, sum(i) over (order by i rows between 1 preceding and 1 following) as sum_rows
FROM generate_series(1, 10) i;
SELECT * FROM v_window;
i | sum_rows
----+----------
1 | 3
2 | 6
3 | 9
4 | 12
5 | 15
6 | 18
7 | 21
8 | 24
9 | 27
10 | 19
(10 rows)
SELECT pg_get_viewdef('v_window');
pg_get_viewdef
---------------------------------------------------------------------------------------
SELECT i.i, +
sum(i.i) OVER (ORDER BY i.i ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING) AS sum_rows+
FROM generate_series(1, 10) i(i);
(1 row)
CREATE OR REPLACE TEMP VIEW v_window AS
SELECT i, sum(i) over (order by i rows between 1 preceding and 1 following
exclude current row) as sum_rows FROM generate_series(1, 10) i;
SELECT * FROM v_window;
i | sum_rows
----+----------
1 | 2
2 | 4
3 | 6
4 | 8
5 | 10
6 | 12
7 | 14
8 | 16
9 | 18
10 | 9
(10 rows)
SELECT pg_get_viewdef('v_window');
pg_get_viewdef
-----------------------------------------------------------------------------------------------------------
SELECT i.i, +
sum(i.i) OVER (ORDER BY i.i ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING EXCLUDE CURRENT ROW) AS sum_rows+
FROM generate_series(1, 10) i(i);
(1 row)
CREATE OR REPLACE TEMP VIEW v_window AS
SELECT i, sum(i) over (order by i rows between 1 preceding and 1 following
exclude group) as sum_rows FROM generate_series(1, 10) i;
SELECT * FROM v_window;
i | sum_rows
----+----------
1 | 2
2 | 4
3 | 6
4 | 8
5 | 10
6 | 12
7 | 14
8 | 16
9 | 18
10 | 9
(10 rows)
SELECT pg_get_viewdef('v_window');
pg_get_viewdef
-----------------------------------------------------------------------------------------------------
SELECT i.i, +
sum(i.i) OVER (ORDER BY i.i ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING EXCLUDE GROUP) AS sum_rows+
FROM generate_series(1, 10) i(i);
(1 row)
CREATE OR REPLACE TEMP VIEW v_window AS
SELECT i, sum(i) over (order by i rows between 1 preceding and 1 following
exclude ties) as sum_rows FROM generate_series(1, 10) i;
SELECT * FROM v_window;
i | sum_rows
----+----------
1 | 3
2 | 6
3 | 9
4 | 12
5 | 15
6 | 18
7 | 21
8 | 24
9 | 27
10 | 19
(10 rows)
SELECT pg_get_viewdef('v_window');
pg_get_viewdef
----------------------------------------------------------------------------------------------------
SELECT i.i, +
sum(i.i) OVER (ORDER BY i.i ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING EXCLUDE TIES) AS sum_rows+
FROM generate_series(1, 10) i(i);
(1 row)
CREATE OR REPLACE TEMP VIEW v_window AS
SELECT i, sum(i) over (order by i rows between 1 preceding and 1 following
exclude no others) as sum_rows FROM generate_series(1, 10) i;
SELECT * FROM v_window;
i | sum_rows
----+----------
1 | 3
2 | 6
3 | 9
4 | 12
5 | 15
6 | 18
7 | 21
8 | 24
9 | 27
10 | 19
(10 rows)
SELECT pg_get_viewdef('v_window');
pg_get_viewdef
---------------------------------------------------------------------------------------
SELECT i.i, +
sum(i.i) OVER (ORDER BY i.i ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING) AS sum_rows+
FROM generate_series(1, 10) i(i);
(1 row)
CREATE OR REPLACE TEMP VIEW v_window AS
SELECT i, sum(i) over (order by i groups between 1 preceding and 1 following) as sum_rows FROM generate_series(1, 10) i;
SELECT * FROM v_window;
i | sum_rows
----+----------
1 | 3
2 | 6
3 | 9
4 | 12
5 | 15
6 | 18
7 | 21
8 | 24
9 | 27
10 | 19
(10 rows)
SELECT pg_get_viewdef('v_window');
pg_get_viewdef
-----------------------------------------------------------------------------------------
SELECT i.i, +
sum(i.i) OVER (ORDER BY i.i GROUPS BETWEEN 1 PRECEDING AND 1 FOLLOWING) AS sum_rows+
FROM generate_series(1, 10) i(i);
(1 row)
DROP VIEW v_window;
CREATE TEMP VIEW v_window AS
SELECT i, min(i) over (order by i range between '1 day' preceding and '10 days' following) as min_i
FROM generate_series(now(), now()+'100 days'::interval, '1 hour') i;
SELECT pg_get_viewdef('v_window');
pg_get_viewdef
---------------------------------------------------------------------------------------------------------------------------
SELECT i.i, +
min(i.i) OVER (ORDER BY i.i RANGE BETWEEN '@ 1 day'::interval PRECEDING AND '@ 10 days'::interval FOLLOWING) AS min_i+
FROM generate_series(now(), (now() + '@ 100 days'::interval), '@ 1 hour'::interval) i(i);
(1 row)
-- RANGE offset PRECEDING/FOLLOWING tests
SELECT sum(unique1) over (order by four range between 2::int8 preceding and 1::int2 preceding),
unique1, four
FROM tenk1 WHERE unique1 < 10;
sum | unique1 | four
-----+---------+------
| 0 | 0
| 8 | 0
| 4 | 0
12 | 5 | 1
12 | 9 | 1
12 | 1 | 1
27 | 6 | 2
27 | 2 | 2
23 | 3 | 3
23 | 7 | 3
(10 rows)
SELECT sum(unique1) over (order by four desc range between 2::int8 preceding and 1::int2 preceding),
unique1, four
FROM tenk1 WHERE unique1 < 10;
sum | unique1 | four
-----+---------+------
| 3 | 3
| 7 | 3
10 | 6 | 2
10 | 2 | 2
18 | 9 | 1
18 | 5 | 1
18 | 1 | 1
23 | 0 | 0
23 | 8 | 0
23 | 4 | 0
(10 rows)
SELECT sum(unique1) over (order by four range between 2::int8 preceding and 1::int2 preceding exclude no others),
unique1, four
FROM tenk1 WHERE unique1 < 10;
sum | unique1 | four
-----+---------+------
| 0 | 0
| 8 | 0
| 4 | 0
12 | 5 | 1
12 | 9 | 1
12 | 1 | 1
27 | 6 | 2
27 | 2 | 2
23 | 3 | 3
23 | 7 | 3
(10 rows)
SELECT sum(unique1) over (order by four range between 2::int8 preceding and 1::int2 preceding exclude current row),
unique1, four
FROM tenk1 WHERE unique1 < 10;
sum | unique1 | four
-----+---------+------
| 0 | 0
| 8 | 0
| 4 | 0
12 | 5 | 1
12 | 9 | 1
12 | 1 | 1
27 | 6 | 2
27 | 2 | 2
23 | 3 | 3
23 | 7 | 3
(10 rows)
SELECT sum(unique1) over (order by four range between 2::int8 preceding and 1::int2 preceding exclude group),
unique1, four
FROM tenk1 WHERE unique1 < 10;
sum | unique1 | four
-----+---------+------
| 0 | 0
| 8 | 0
| 4 | 0
12 | 5 | 1
12 | 9 | 1
12 | 1 | 1
27 | 6 | 2
27 | 2 | 2
23 | 3 | 3
23 | 7 | 3
(10 rows)
SELECT sum(unique1) over (order by four range between 2::int8 preceding and 1::int2 preceding exclude ties),
unique1, four
FROM tenk1 WHERE unique1 < 10;
sum | unique1 | four
-----+---------+------
| 0 | 0
| 8 | 0
| 4 | 0
12 | 5 | 1
12 | 9 | 1
12 | 1 | 1
27 | 6 | 2
27 | 2 | 2
23 | 3 | 3
23 | 7 | 3
(10 rows)
SELECT sum(unique1) over (order by four range between 2::int8 preceding and 6::int2 following exclude ties),
unique1, four
FROM tenk1 WHERE unique1 < 10;
sum | unique1 | four
-----+---------+------
33 | 0 | 0
41 | 8 | 0
37 | 4 | 0
35 | 5 | 1
39 | 9 | 1
31 | 1 | 1
43 | 6 | 2
39 | 2 | 2
26 | 3 | 3
30 | 7 | 3
(10 rows)
SELECT sum(unique1) over (order by four range between 2::int8 preceding and 6::int2 following exclude group),
unique1, four
FROM tenk1 WHERE unique1 < 10;
sum | unique1 | four
-----+---------+------
33 | 0 | 0
33 | 8 | 0
33 | 4 | 0
30 | 5 | 1
30 | 9 | 1
30 | 1 | 1
37 | 6 | 2
37 | 2 | 2
23 | 3 | 3
23 | 7 | 3
(10 rows)
SELECT sum(unique1) over (partition by four order by unique1 range between 5::int8 preceding and 6::int2 following),
unique1, four
FROM tenk1 WHERE unique1 < 10;
sum | unique1 | four
-----+---------+------
4 | 0 | 0
12 | 4 | 0
12 | 8 | 0
6 | 1 | 1
15 | 5 | 1
14 | 9 | 1
8 | 2 | 2
8 | 6 | 2
10 | 3 | 3
10 | 7 | 3
(10 rows)
SELECT sum(unique1) over (partition by four order by unique1 range between 5::int8 preceding and 6::int2 following
exclude current row),unique1, four
FROM tenk1 WHERE unique1 < 10;
sum | unique1 | four
-----+---------+------
4 | 0 | 0
8 | 4 | 0
4 | 8 | 0
5 | 1 | 1
10 | 5 | 1
5 | 9 | 1
6 | 2 | 2
2 | 6 | 2
7 | 3 | 3
3 | 7 | 3
(10 rows)
select sum(salary) over (order by enroll_date range between '1 year'::interval preceding and '1 year'::interval following),
salary, enroll_date from empsalary;
sum | salary | enroll_date
-------+--------+-------------
34900 | 5000 | 10-01-2006
34900 | 6000 | 10-01-2006
38400 | 3900 | 12-23-2006
47100 | 4800 | 08-01-2007
47100 | 5200 | 08-01-2007
47100 | 4800 | 08-08-2007
47100 | 5200 | 08-15-2007
36100 | 3500 | 12-10-2007
32200 | 4500 | 01-01-2008
32200 | 4200 | 01-01-2008
(10 rows)
select sum(salary) over (order by enroll_date desc range between '1 year'::interval preceding and '1 year'::interval following),
salary, enroll_date from empsalary;
sum | salary | enroll_date
-------+--------+-------------
32200 | 4200 | 01-01-2008
32200 | 4500 | 01-01-2008
36100 | 3500 | 12-10-2007
47100 | 5200 | 08-15-2007
47100 | 4800 | 08-08-2007
47100 | 4800 | 08-01-2007
47100 | 5200 | 08-01-2007
38400 | 3900 | 12-23-2006
34900 | 5000 | 10-01-2006
34900 | 6000 | 10-01-2006
(10 rows)
select sum(salary) over (order by enroll_date desc range between '1 year'::interval following and '1 year'::interval following),
salary, enroll_date from empsalary;
sum | salary | enroll_date
-----+--------+-------------
| 4200 | 01-01-2008
| 4500 | 01-01-2008
| 3500 | 12-10-2007
| 5200 | 08-15-2007
| 4800 | 08-08-2007
| 4800 | 08-01-2007
| 5200 | 08-01-2007
| 3900 | 12-23-2006
| 5000 | 10-01-2006
| 6000 | 10-01-2006
(10 rows)
select sum(salary) over (order by enroll_date range between '1 year'::interval preceding and '1 year'::interval following
exclude current row), salary, enroll_date from empsalary;
sum | salary | enroll_date
-------+--------+-------------
29900 | 5000 | 10-01-2006
28900 | 6000 | 10-01-2006
34500 | 3900 | 12-23-2006
42300 | 4800 | 08-01-2007
41900 | 5200 | 08-01-2007
42300 | 4800 | 08-08-2007
41900 | 5200 | 08-15-2007
32600 | 3500 | 12-10-2007
27700 | 4500 | 01-01-2008
28000 | 4200 | 01-01-2008
(10 rows)
select sum(salary) over (order by enroll_date range between '1 year'::interval preceding and '1 year'::interval following
exclude group), salary, enroll_date from empsalary;
sum | salary | enroll_date
-------+--------+-------------
23900 | 5000 | 10-01-2006
23900 | 6000 | 10-01-2006
34500 | 3900 | 12-23-2006
37100 | 4800 | 08-01-2007
37100 | 5200 | 08-01-2007
42300 | 4800 | 08-08-2007
41900 | 5200 | 08-15-2007
32600 | 3500 | 12-10-2007
23500 | 4500 | 01-01-2008
23500 | 4200 | 01-01-2008
(10 rows)
select sum(salary) over (order by enroll_date range between '1 year'::interval preceding and '1 year'::interval following
exclude ties), salary, enroll_date from empsalary;
sum | salary | enroll_date
-------+--------+-------------
28900 | 5000 | 10-01-2006
29900 | 6000 | 10-01-2006
38400 | 3900 | 12-23-2006
41900 | 4800 | 08-01-2007
42300 | 5200 | 08-01-2007
47100 | 4800 | 08-08-2007
47100 | 5200 | 08-15-2007
36100 | 3500 | 12-10-2007
28000 | 4500 | 01-01-2008
27700 | 4200 | 01-01-2008
(10 rows)
select first_value(salary) over(order by salary range between 1000 preceding and 1000 following),
lead(salary) over(order by salary range between 1000 preceding and 1000 following),
nth_value(salary, 1) over(order by salary range between 1000 preceding and 1000 following),
salary from empsalary;
first_value | lead | nth_value | salary
-------------+------+-----------+--------
3500 | 3900 | 3500 | 3500
3500 | 4200 | 3500 | 3900
3500 | 4500 | 3500 | 4200
3500 | 4800 | 3500 | 4500
3900 | 4800 | 3900 | 4800
3900 | 5000 | 3900 | 4800
4200 | 5200 | 4200 | 5000
4200 | 5200 | 4200 | 5200
4200 | 6000 | 4200 | 5200
5000 | | 5000 | 6000
(10 rows)
select last_value(salary) over(order by salary range between 1000 preceding and 1000 following),
lag(salary) over(order by salary range between 1000 preceding and 1000 following),
salary from empsalary;
last_value | lag | salary
------------+------+--------
4500 | | 3500
4800 | 3500 | 3900
5200 | 3900 | 4200
5200 | 4200 | 4500
5200 | 4500 | 4800
5200 | 4800 | 4800
6000 | 4800 | 5000
6000 | 5000 | 5200
6000 | 5200 | 5200
6000 | 5200 | 6000
(10 rows)
select first_value(salary) over(order by salary range between 1000 following and 3000 following
exclude current row),
lead(salary) over(order by salary range between 1000 following and 3000 following exclude ties),
nth_value(salary, 1) over(order by salary range between 1000 following and 3000 following
exclude ties),
salary from empsalary;
first_value | lead | nth_value | salary
-------------+------+-----------+--------
4500 | 3900 | 4500 | 3500
5000 | 4200 | 5000 | 3900
5200 | 4500 | 5200 | 4200
6000 | 4800 | 6000 | 4500
6000 | 4800 | 6000 | 4800
6000 | 5000 | 6000 | 4800
6000 | 5200 | 6000 | 5000
| 5200 | | 5200
| 6000 | | 5200
| | | 6000
(10 rows)
select last_value(salary) over(order by salary range between 1000 following and 3000 following
exclude group),
lag(salary) over(order by salary range between 1000 following and 3000 following exclude group),
salary from empsalary;
last_value | lag | salary
------------+------+--------
6000 | | 3500
6000 | 3500 | 3900
6000 | 3900 | 4200
6000 | 4200 | 4500
6000 | 4500 | 4800
6000 | 4800 | 4800
6000 | 4800 | 5000
| 5000 | 5200
| 5200 | 5200
| 5200 | 6000
(10 rows)
select first_value(salary) over(order by enroll_date range between unbounded preceding and '1 year'::interval following
exclude ties),
last_value(salary) over(order by enroll_date range between unbounded preceding and '1 year'::interval following),
salary, enroll_date from empsalary;
first_value | last_value | salary | enroll_date
-------------+------------+--------+-------------
5000 | 5200 | 5000 | 10-01-2006
6000 | 5200 | 6000 | 10-01-2006
5000 | 3500 | 3900 | 12-23-2006
5000 | 4200 | 4800 | 08-01-2007
5000 | 4200 | 5200 | 08-01-2007
5000 | 4200 | 4800 | 08-08-2007
5000 | 4200 | 5200 | 08-15-2007
5000 | 4200 | 3500 | 12-10-2007
5000 | 4200 | 4500 | 01-01-2008
5000 | 4200 | 4200 | 01-01-2008
(10 rows)
select first_value(salary) over(order by enroll_date range between unbounded preceding and '1 year'::interval following
exclude ties),
last_value(salary) over(order by enroll_date range between unbounded preceding and '1 year'::interval following
exclude ties),
salary, enroll_date from empsalary;
first_value | last_value | salary | enroll_date
-------------+------------+--------+-------------
5000 | 5200 | 5000 | 10-01-2006
6000 | 5200 | 6000 | 10-01-2006
5000 | 3500 | 3900 | 12-23-2006
5000 | 4200 | 4800 | 08-01-2007
5000 | 4200 | 5200 | 08-01-2007
5000 | 4200 | 4800 | 08-08-2007
5000 | 4200 | 5200 | 08-15-2007
5000 | 4200 | 3500 | 12-10-2007
5000 | 4500 | 4500 | 01-01-2008
5000 | 4200 | 4200 | 01-01-2008
(10 rows)
select first_value(salary) over(order by enroll_date range between unbounded preceding and '1 year'::interval following
exclude group),
last_value(salary) over(order by enroll_date range between unbounded preceding and '1 year'::interval following
exclude group),
salary, enroll_date from empsalary;
first_value | last_value | salary | enroll_date
-------------+------------+--------+-------------
3900 | 5200 | 5000 | 10-01-2006
3900 | 5200 | 6000 | 10-01-2006
5000 | 3500 | 3900 | 12-23-2006
5000 | 4200 | 4800 | 08-01-2007
5000 | 4200 | 5200 | 08-01-2007
5000 | 4200 | 4800 | 08-08-2007
5000 | 4200 | 5200 | 08-15-2007
5000 | 4200 | 3500 | 12-10-2007
5000 | 3500 | 4500 | 01-01-2008
5000 | 3500 | 4200 | 01-01-2008
(10 rows)
select first_value(salary) over(order by enroll_date range between unbounded preceding and '1 year'::interval following
exclude current row),
last_value(salary) over(order by enroll_date range between unbounded preceding and '1 year'::interval following
exclude current row),
salary, enroll_date from empsalary;
first_value | last_value | salary | enroll_date
-------------+------------+--------+-------------
6000 | 5200 | 5000 | 10-01-2006
5000 | 5200 | 6000 | 10-01-2006
5000 | 3500 | 3900 | 12-23-2006
5000 | 4200 | 4800 | 08-01-2007
5000 | 4200 | 5200 | 08-01-2007
5000 | 4200 | 4800 | 08-08-2007
5000 | 4200 | 5200 | 08-15-2007
5000 | 4200 | 3500 | 12-10-2007
5000 | 4200 | 4500 | 01-01-2008
5000 | 4500 | 4200 | 01-01-2008
(10 rows)
-- RANGE offset PRECEDING/FOLLOWING with null values
select x, y,
first_value(y) over w,
last_value(y) over w
from
(select x, x as y from generate_series(1,5) as x
union all select null, 42
union all select null, 43) ss
window w as
(order by x asc nulls first range between 2 preceding and 2 following);
x | y | first_value | last_value
---+----+-------------+------------
| 42 | 42 | 43
| 43 | 42 | 43
1 | 1 | 1 | 3
2 | 2 | 1 | 4
3 | 3 | 1 | 5
4 | 4 | 2 | 5
5 | 5 | 3 | 5
(7 rows)
select x, y,
first_value(y) over w,
last_value(y) over w
from
(select x, x as y from generate_series(1,5) as x
union all select null, 42
union all select null, 43) ss
window w as
(order by x asc nulls last range between 2 preceding and 2 following);
x | y | first_value | last_value
---+----+-------------+------------
1 | 1 | 1 | 3
2 | 2 | 1 | 4
3 | 3 | 1 | 5
4 | 4 | 2 | 5
5 | 5 | 3 | 5
| 42 | 42 | 43
| 43 | 42 | 43
(7 rows)
select x, y,
first_value(y) over w,
last_value(y) over w
from
(select x, x as y from generate_series(1,5) as x
union all select null, 42
union all select null, 43) ss
window w as
(order by x desc nulls first range between 2 preceding and 2 following);
x | y | first_value | last_value
---+----+-------------+------------
| 43 | 43 | 42
| 42 | 43 | 42
5 | 5 | 5 | 3
4 | 4 | 5 | 2
3 | 3 | 5 | 1
2 | 2 | 4 | 1
1 | 1 | 3 | 1
(7 rows)
select x, y,
first_value(y) over w,
last_value(y) over w
from
(select x, x as y from generate_series(1,5) as x
union all select null, 42
union all select null, 43) ss
window w as
(order by x desc nulls last range between 2 preceding and 2 following);
x | y | first_value | last_value
---+----+-------------+------------
5 | 5 | 5 | 3
4 | 4 | 5 | 2
3 | 3 | 5 | 1
2 | 2 | 4 | 1
1 | 1 | 3 | 1
| 42 | 42 | 43
| 43 | 42 | 43
(7 rows)
-- Check overflow behavior for various integer sizes
select x, last_value(x) over (order by x::smallint range between current row and 2147450884 following)
from generate_series(32764, 32766) x;
x | last_value
-------+------------
32764 | 32766
32765 | 32766
32766 | 32766
(3 rows)
select x, last_value(x) over (order by x::smallint desc range between current row and 2147450885 following)
from generate_series(-32766, -32764) x;
x | last_value
--------+------------
-32764 | -32766
-32765 | -32766
-32766 | -32766
(3 rows)
select x, last_value(x) over (order by x range between current row and 4 following)
from generate_series(2147483644, 2147483646) x;
x | last_value
------------+------------
2147483644 | 2147483646
2147483645 | 2147483646
2147483646 | 2147483646
(3 rows)
select x, last_value(x) over (order by x desc range between current row and 5 following)
from generate_series(-2147483646, -2147483644) x;
x | last_value
-------------+-------------
-2147483644 | -2147483646
-2147483645 | -2147483646
-2147483646 | -2147483646
(3 rows)
select x, last_value(x) over (order by x range between current row and 4 following)
from generate_series(9223372036854775804, 9223372036854775806) x;
x | last_value
---------------------+---------------------
9223372036854775804 | 9223372036854775806
9223372036854775805 | 9223372036854775806
9223372036854775806 | 9223372036854775806
(3 rows)
select x, last_value(x) over (order by x desc range between current row and 5 following)
from generate_series(-9223372036854775806, -9223372036854775804) x;
x | last_value
----------------------+----------------------
-9223372036854775804 | -9223372036854775806
-9223372036854775805 | -9223372036854775806
-9223372036854775806 | -9223372036854775806
(3 rows)
-- Test in_range for other numeric datatypes
create temp table numerics(
id int,
f_float4 float4,
f_float8 float8,
f_numeric numeric
);
insert into numerics values
(0, '-infinity', '-infinity', '-1000'), -- numeric type lacks infinities
(1, -3, -3, -3),
(2, -1, -1, -1),
(3, 0, 0, 0),
(4, 1.1, 1.1, 1.1),
(5, 1.12, 1.12, 1.12),
(6, 2, 2, 2),
(7, 100, 100, 100),
(8, 'infinity', 'infinity', '1000'),
(9, 'NaN', 'NaN', 'NaN');
select id, f_float4, first_value(id) over w, last_value(id) over w
from numerics
window w as (order by f_float4 range between
1 preceding and 1 following);
id | f_float4 | first_value | last_value
----+-----------+-------------+------------
0 | -Infinity | 0 | 0
1 | -3 | 1 | 1
2 | -1 | 2 | 3
3 | 0 | 2 | 3
4 | 1.1 | 4 | 6
5 | 1.12 | 4 | 6
6 | 2 | 4 | 6
7 | 100 | 7 | 7
8 | Infinity | 8 | 8
9 | NaN | 9 | 9
(10 rows)
select id, f_float4, first_value(id) over w, last_value(id) over w
from numerics
window w as (order by f_float4 range between
1 preceding and 1.1::float4 following);
id | f_float4 | first_value | last_value
----+-----------+-------------+------------
0 | -Infinity | 0 | 0
1 | -3 | 1 | 1
2 | -1 | 2 | 3
3 | 0 | 2 | 4
4 | 1.1 | 4 | 6
5 | 1.12 | 4 | 6
6 | 2 | 4 | 6
7 | 100 | 7 | 7
8 | Infinity | 8 | 8
9 | NaN | 9 | 9
(10 rows)
select id, f_float4, first_value(id) over w, last_value(id) over w
from numerics
window w as (order by f_float4 range between
'inf' preceding and 'inf' following);
id | f_float4 | first_value | last_value
----+-----------+-------------+------------
0 | -Infinity | 0 | 8
1 | -3 | 0 | 8
2 | -1 | 0 | 8
3 | 0 | 0 | 8
4 | 1.1 | 0 | 8
5 | 1.12 | 0 | 8
6 | 2 | 0 | 8
7 | 100 | 0 | 8
8 | Infinity | 0 | 8
9 | NaN | 9 | 9
(10 rows)
select id, f_float4, first_value(id) over w, last_value(id) over w
from numerics
window w as (order by f_float4 range between
1.1 preceding and 'NaN' following); -- error, NaN disallowed
ERROR: invalid preceding or following size in window function
select id, f_float8, first_value(id) over w, last_value(id) over w
from numerics
window w as (order by f_float8 range between
1 preceding and 1 following);
id | f_float8 | first_value | last_value
----+-----------+-------------+------------
0 | -Infinity | 0 | 0
1 | -3 | 1 | 1
2 | -1 | 2 | 3
3 | 0 | 2 | 3
4 | 1.1 | 4 | 6
5 | 1.12 | 4 | 6
6 | 2 | 4 | 6
7 | 100 | 7 | 7
8 | Infinity | 8 | 8
9 | NaN | 9 | 9
(10 rows)
select id, f_float8, first_value(id) over w, last_value(id) over w
from numerics
window w as (order by f_float8 range between
1 preceding and 1.1::float8 following);
id | f_float8 | first_value | last_value
----+-----------+-------------+------------
0 | -Infinity | 0 | 0
1 | -3 | 1 | 1
2 | -1 | 2 | 3
3 | 0 | 2 | 4
4 | 1.1 | 4 | 6
5 | 1.12 | 4 | 6
6 | 2 | 4 | 6
7 | 100 | 7 | 7
8 | Infinity | 8 | 8
9 | NaN | 9 | 9
(10 rows)
select id, f_float8, first_value(id) over w, last_value(id) over w
from numerics
window w as (order by f_float8 range between
'inf' preceding and 'inf' following);
id | f_float8 | first_value | last_value
----+-----------+-------------+------------
0 | -Infinity | 0 | 8
1 | -3 | 0 | 8
2 | -1 | 0 | 8
3 | 0 | 0 | 8
4 | 1.1 | 0 | 8
5 | 1.12 | 0 | 8
6 | 2 | 0 | 8
7 | 100 | 0 | 8
8 | Infinity | 0 | 8
9 | NaN | 9 | 9
(10 rows)
select id, f_float8, first_value(id) over w, last_value(id) over w
from numerics
window w as (order by f_float8 range between
1.1 preceding and 'NaN' following); -- error, NaN disallowed
ERROR: invalid preceding or following size in window function
select id, f_numeric, first_value(id) over w, last_value(id) over w
from numerics
window w as (order by f_numeric range between
1 preceding and 1 following);
id | f_numeric | first_value | last_value
----+-----------+-------------+------------
0 | -1000 | 0 | 0
1 | -3 | 1 | 1
2 | -1 | 2 | 3
3 | 0 | 2 | 3
4 | 1.1 | 4 | 6
5 | 1.12 | 4 | 6
6 | 2 | 4 | 6
7 | 100 | 7 | 7
8 | 1000 | 8 | 8
9 | NaN | 9 | 9
(10 rows)
select id, f_numeric, first_value(id) over w, last_value(id) over w
from numerics
window w as (order by f_numeric range between
1 preceding and 1.1::numeric following);
id | f_numeric | first_value | last_value
----+-----------+-------------+------------
0 | -1000 | 0 | 0
1 | -3 | 1 | 1
2 | -1 | 2 | 3
3 | 0 | 2 | 4
4 | 1.1 | 4 | 6
5 | 1.12 | 4 | 6
6 | 2 | 4 | 6
7 | 100 | 7 | 7
8 | 1000 | 8 | 8
9 | NaN | 9 | 9
(10 rows)
select id, f_numeric, first_value(id) over w, last_value(id) over w
from numerics
window w as (order by f_numeric range between
1 preceding and 1.1::float8 following); -- currently unsupported
ERROR: RANGE with offset PRECEDING/FOLLOWING is not supported for column type numeric and offset type double precision
LINE 4: 1 preceding and 1.1::float8 following);
^
HINT: Cast the offset value to an appropriate type.
select id, f_numeric, first_value(id) over w, last_value(id) over w
from numerics
window w as (order by f_numeric range between
1.1 preceding and 'NaN' following); -- error, NaN disallowed
ERROR: invalid preceding or following size in window function
-- Test in_range for other datetime datatypes
create temp table datetimes(
id int,
f_time time,
f_timetz timetz,
f_interval interval,
f_timestamptz timestamptz,
f_timestamp timestamp
);
insert into datetimes values
(1, '11:00', '11:00 BST', '1 year', '2000-10-19 10:23:54+01', '2000-10-19 10:23:54'),
(2, '12:00', '12:00 BST', '2 years', '2001-10-19 10:23:54+01', '2001-10-19 10:23:54'),
(3, '13:00', '13:00 BST', '3 years', '2001-10-19 10:23:54+01', '2001-10-19 10:23:54'),
(4, '14:00', '14:00 BST', '4 years', '2002-10-19 10:23:54+01', '2002-10-19 10:23:54'),
(5, '15:00', '15:00 BST', '5 years', '2003-10-19 10:23:54+01', '2003-10-19 10:23:54'),
(6, '15:00', '15:00 BST', '5 years', '2004-10-19 10:23:54+01', '2004-10-19 10:23:54'),
(7, '17:00', '17:00 BST', '7 years', '2005-10-19 10:23:54+01', '2005-10-19 10:23:54'),
(8, '18:00', '18:00 BST', '8 years', '2006-10-19 10:23:54+01', '2006-10-19 10:23:54'),
(9, '19:00', '19:00 BST', '9 years', '2007-10-19 10:23:54+01', '2007-10-19 10:23:54'),
(10, '20:00', '20:00 BST', '10 years', '2008-10-19 10:23:54+01', '2008-10-19 10:23:54');
select id, f_time, first_value(id) over w, last_value(id) over w
from datetimes
window w as (order by f_time range between
'70 min'::interval preceding and '2 hours'::interval following);
id | f_time | first_value | last_value
----+----------+-------------+------------
1 | 11:00:00 | 1 | 3
2 | 12:00:00 | 1 | 4
3 | 13:00:00 | 2 | 6
4 | 14:00:00 | 3 | 6
5 | 15:00:00 | 4 | 7
6 | 15:00:00 | 4 | 7
7 | 17:00:00 | 7 | 9
8 | 18:00:00 | 7 | 10
9 | 19:00:00 | 8 | 10
10 | 20:00:00 | 9 | 10
(10 rows)
select id, f_time, first_value(id) over w, last_value(id) over w
from datetimes
window w as (order by f_time desc range between
'70 min' preceding and '2 hours' following);
id | f_time | first_value | last_value
----+----------+-------------+------------
10 | 20:00:00 | 10 | 8
9 | 19:00:00 | 10 | 7
8 | 18:00:00 | 9 | 7
7 | 17:00:00 | 8 | 5
6 | 15:00:00 | 6 | 3
5 | 15:00:00 | 6 | 3
4 | 14:00:00 | 6 | 2
3 | 13:00:00 | 4 | 1
2 | 12:00:00 | 3 | 1
1 | 11:00:00 | 2 | 1
(10 rows)
select id, f_timetz, first_value(id) over w, last_value(id) over w
from datetimes
window w as (order by f_timetz range between
'70 min'::interval preceding and '2 hours'::interval following);
id | f_timetz | first_value | last_value
----+-------------+-------------+------------
1 | 11:00:00+01 | 1 | 3
2 | 12:00:00+01 | 1 | 4
3 | 13:00:00+01 | 2 | 6
4 | 14:00:00+01 | 3 | 6
5 | 15:00:00+01 | 4 | 7
6 | 15:00:00+01 | 4 | 7
7 | 17:00:00+01 | 7 | 9
8 | 18:00:00+01 | 7 | 10
9 | 19:00:00+01 | 8 | 10
10 | 20:00:00+01 | 9 | 10
(10 rows)
select id, f_timetz, first_value(id) over w, last_value(id) over w
from datetimes
window w as (order by f_timetz desc range between
'70 min' preceding and '2 hours' following);
id | f_timetz | first_value | last_value
----+-------------+-------------+------------
10 | 20:00:00+01 | 10 | 8
9 | 19:00:00+01 | 10 | 7
8 | 18:00:00+01 | 9 | 7
7 | 17:00:00+01 | 8 | 5
6 | 15:00:00+01 | 6 | 3
5 | 15:00:00+01 | 6 | 3
4 | 14:00:00+01 | 6 | 2
3 | 13:00:00+01 | 4 | 1
2 | 12:00:00+01 | 3 | 1
1 | 11:00:00+01 | 2 | 1
(10 rows)
select id, f_interval, first_value(id) over w, last_value(id) over w
from datetimes
window w as (order by f_interval range between
'1 year'::interval preceding and '1 year'::interval following);
id | f_interval | first_value | last_value
----+------------+-------------+------------
1 | @ 1 year | 1 | 2
2 | @ 2 years | 1 | 3
3 | @ 3 years | 2 | 4
4 | @ 4 years | 3 | 6
5 | @ 5 years | 4 | 6
6 | @ 5 years | 4 | 6
7 | @ 7 years | 7 | 8
8 | @ 8 years | 7 | 9
9 | @ 9 years | 8 | 10
10 | @ 10 years | 9 | 10
(10 rows)
select id, f_interval, first_value(id) over w, last_value(id) over w
from datetimes
window w as (order by f_interval desc range between
'1 year' preceding and '1 year' following);
id | f_interval | first_value | last_value
----+------------+-------------+------------
10 | @ 10 years | 10 | 9
9 | @ 9 years | 10 | 8
8 | @ 8 years | 9 | 7
7 | @ 7 years | 8 | 7
6 | @ 5 years | 6 | 4
5 | @ 5 years | 6 | 4
4 | @ 4 years | 6 | 3
3 | @ 3 years | 4 | 2
2 | @ 2 years | 3 | 1
1 | @ 1 year | 2 | 1
(10 rows)
select id, f_timestamptz, first_value(id) over w, last_value(id) over w
from datetimes
window w as (order by f_timestamptz range between
'1 year'::interval preceding and '1 year'::interval following);
id | f_timestamptz | first_value | last_value
----+------------------------------+-------------+------------
1 | Thu Oct 19 02:23:54 2000 PDT | 1 | 3
2 | Fri Oct 19 02:23:54 2001 PDT | 1 | 4
3 | Fri Oct 19 02:23:54 2001 PDT | 1 | 4
4 | Sat Oct 19 02:23:54 2002 PDT | 2 | 5
5 | Sun Oct 19 02:23:54 2003 PDT | 4 | 6
6 | Tue Oct 19 02:23:54 2004 PDT | 5 | 7
7 | Wed Oct 19 02:23:54 2005 PDT | 6 | 8
8 | Thu Oct 19 02:23:54 2006 PDT | 7 | 9
9 | Fri Oct 19 02:23:54 2007 PDT | 8 | 10
10 | Sun Oct 19 02:23:54 2008 PDT | 9 | 10
(10 rows)
select id, f_timestamptz, first_value(id) over w, last_value(id) over w
from datetimes
window w as (order by f_timestamptz desc range between
'1 year' preceding and '1 year' following);
id | f_timestamptz | first_value | last_value
----+------------------------------+-------------+------------
10 | Sun Oct 19 02:23:54 2008 PDT | 10 | 9
9 | Fri Oct 19 02:23:54 2007 PDT | 10 | 8
8 | Thu Oct 19 02:23:54 2006 PDT | 9 | 7
7 | Wed Oct 19 02:23:54 2005 PDT | 8 | 6
6 | Tue Oct 19 02:23:54 2004 PDT | 7 | 5
5 | Sun Oct 19 02:23:54 2003 PDT | 6 | 4
4 | Sat Oct 19 02:23:54 2002 PDT | 5 | 2
3 | Fri Oct 19 02:23:54 2001 PDT | 4 | 1
2 | Fri Oct 19 02:23:54 2001 PDT | 4 | 1
1 | Thu Oct 19 02:23:54 2000 PDT | 3 | 1
(10 rows)
select id, f_timestamp, first_value(id) over w, last_value(id) over w
from datetimes
window w as (order by f_timestamp range between
'1 year'::interval preceding and '1 year'::interval following);
id | f_timestamp | first_value | last_value
----+--------------------------+-------------+------------
1 | Thu Oct 19 10:23:54 2000 | 1 | 3
2 | Fri Oct 19 10:23:54 2001 | 1 | 4
3 | Fri Oct 19 10:23:54 2001 | 1 | 4
4 | Sat Oct 19 10:23:54 2002 | 2 | 5
5 | Sun Oct 19 10:23:54 2003 | 4 | 6
6 | Tue Oct 19 10:23:54 2004 | 5 | 7
7 | Wed Oct 19 10:23:54 2005 | 6 | 8
8 | Thu Oct 19 10:23:54 2006 | 7 | 9
9 | Fri Oct 19 10:23:54 2007 | 8 | 10
10 | Sun Oct 19 10:23:54 2008 | 9 | 10
(10 rows)
select id, f_timestamp, first_value(id) over w, last_value(id) over w
from datetimes
window w as (order by f_timestamp desc range between
'1 year' preceding and '1 year' following);
id | f_timestamp | first_value | last_value
----+--------------------------+-------------+------------
10 | Sun Oct 19 10:23:54 2008 | 10 | 9
9 | Fri Oct 19 10:23:54 2007 | 10 | 8
8 | Thu Oct 19 10:23:54 2006 | 9 | 7
7 | Wed Oct 19 10:23:54 2005 | 8 | 6
6 | Tue Oct 19 10:23:54 2004 | 7 | 5
5 | Sun Oct 19 10:23:54 2003 | 6 | 4
4 | Sat Oct 19 10:23:54 2002 | 5 | 2
3 | Fri Oct 19 10:23:54 2001 | 4 | 1
2 | Fri Oct 19 10:23:54 2001 | 4 | 1
1 | Thu Oct 19 10:23:54 2000 | 3 | 1
(10 rows)
-- RANGE offset PRECEDING/FOLLOWING error cases
select sum(salary) over (order by enroll_date, salary range between '1 year'::interval preceding and '2 years'::interval following
exclude ties), salary, enroll_date from empsalary;
ERROR: RANGE with offset PRECEDING/FOLLOWING requires exactly one ORDER BY column
LINE 1: select sum(salary) over (order by enroll_date, salary range ...
^
select sum(salary) over (range between '1 year'::interval preceding and '2 years'::interval following
exclude ties), salary, enroll_date from empsalary;
ERROR: RANGE with offset PRECEDING/FOLLOWING requires exactly one ORDER BY column
LINE 1: select sum(salary) over (range between '1 year'::interval pr...
^
select sum(salary) over (order by depname range between '1 year'::interval preceding and '2 years'::interval following
exclude ties), salary, enroll_date from empsalary;
ERROR: RANGE with offset PRECEDING/FOLLOWING is not supported for column type text
LINE 1: ... sum(salary) over (order by depname range between '1 year'::...
^
select max(enroll_date) over (order by enroll_date range between 1 preceding and 2 following
exclude ties), salary, enroll_date from empsalary;
ERROR: RANGE with offset PRECEDING/FOLLOWING is not supported for column type date and offset type integer
LINE 1: ...ll_date) over (order by enroll_date range between 1 precedin...
^
HINT: Cast the offset value to an appropriate type.
select max(enroll_date) over (order by salary range between -1 preceding and 2 following
exclude ties), salary, enroll_date from empsalary;
ERROR: invalid preceding or following size in window function
select max(enroll_date) over (order by salary range between 1 preceding and -2 following
exclude ties), salary, enroll_date from empsalary;
ERROR: invalid preceding or following size in window function
select max(enroll_date) over (order by salary range between '1 year'::interval preceding and '2 years'::interval following
exclude ties), salary, enroll_date from empsalary;
ERROR: RANGE with offset PRECEDING/FOLLOWING is not supported for column type integer and offset type interval
LINE 1: ...(enroll_date) over (order by salary range between '1 year'::...
^
HINT: Cast the offset value to an appropriate type.
select max(enroll_date) over (order by enroll_date range between '1 year'::interval preceding and '-2 years'::interval following
exclude ties), salary, enroll_date from empsalary;
ERROR: invalid preceding or following size in window function
-- GROUPS tests
SELECT sum(unique1) over (order by four groups between unbounded preceding and current row),
unique1, four
FROM tenk1 WHERE unique1 < 10;
sum | unique1 | four
-----+---------+------
12 | 0 | 0
12 | 8 | 0
12 | 4 | 0
27 | 5 | 1
27 | 9 | 1
27 | 1 | 1
35 | 6 | 2
35 | 2 | 2
45 | 3 | 3
45 | 7 | 3
(10 rows)
SELECT sum(unique1) over (order by four groups between unbounded preceding and unbounded following),
unique1, four
FROM tenk1 WHERE unique1 < 10;
sum | unique1 | four
-----+---------+------
45 | 0 | 0
45 | 8 | 0
45 | 4 | 0
45 | 5 | 1
45 | 9 | 1
45 | 1 | 1
45 | 6 | 2
45 | 2 | 2
45 | 3 | 3
45 | 7 | 3
(10 rows)
SELECT sum(unique1) over (order by four groups between current row and unbounded following),
unique1, four
FROM tenk1 WHERE unique1 < 10;
sum | unique1 | four
-----+---------+------
45 | 0 | 0
45 | 8 | 0
45 | 4 | 0
33 | 5 | 1
33 | 9 | 1
33 | 1 | 1
18 | 6 | 2
18 | 2 | 2
10 | 3 | 3
10 | 7 | 3
(10 rows)
SELECT sum(unique1) over (order by four groups between 1 preceding and unbounded following),
unique1, four
FROM tenk1 WHERE unique1 < 10;
sum | unique1 | four
-----+---------+------
45 | 0 | 0
45 | 8 | 0
45 | 4 | 0
45 | 5 | 1
45 | 9 | 1
45 | 1 | 1
33 | 6 | 2
33 | 2 | 2
18 | 3 | 3
18 | 7 | 3
(10 rows)
SELECT sum(unique1) over (order by four groups between 1 following and unbounded following),
unique1, four
FROM tenk1 WHERE unique1 < 10;
sum | unique1 | four
-----+---------+------
33 | 0 | 0
33 | 8 | 0
33 | 4 | 0
18 | 5 | 1
18 | 9 | 1
18 | 1 | 1
10 | 6 | 2
10 | 2 | 2
| 3 | 3
| 7 | 3
(10 rows)
SELECT sum(unique1) over (order by four groups between unbounded preceding and 2 following),
unique1, four
FROM tenk1 WHERE unique1 < 10;
sum | unique1 | four
-----+---------+------
35 | 0 | 0
35 | 8 | 0
35 | 4 | 0
45 | 5 | 1
45 | 9 | 1
45 | 1 | 1
45 | 6 | 2
45 | 2 | 2
45 | 3 | 3
45 | 7 | 3
(10 rows)
SELECT sum(unique1) over (order by four groups between 2 preceding and 1 preceding),
unique1, four
FROM tenk1 WHERE unique1 < 10;
sum | unique1 | four
-----+---------+------
| 0 | 0
| 8 | 0
| 4 | 0
12 | 5 | 1
12 | 9 | 1
12 | 1 | 1
27 | 6 | 2
27 | 2 | 2
23 | 3 | 3
23 | 7 | 3
(10 rows)
SELECT sum(unique1) over (order by four groups between 2 preceding and 1 following),
unique1, four
FROM tenk1 WHERE unique1 < 10;
sum | unique1 | four
-----+---------+------
27 | 0 | 0
27 | 8 | 0
27 | 4 | 0
35 | 5 | 1
35 | 9 | 1
35 | 1 | 1
45 | 6 | 2
45 | 2 | 2
33 | 3 | 3
33 | 7 | 3
(10 rows)
SELECT sum(unique1) over (order by four groups between 0 preceding and 0 following),
unique1, four
FROM tenk1 WHERE unique1 < 10;
sum | unique1 | four
-----+---------+------
12 | 0 | 0
12 | 8 | 0
12 | 4 | 0
15 | 5 | 1
15 | 9 | 1
15 | 1 | 1
8 | 6 | 2
8 | 2 | 2
10 | 3 | 3
10 | 7 | 3
(10 rows)
SELECT sum(unique1) over (order by four groups between 2 preceding and 1 following
exclude current row), unique1, four
FROM tenk1 WHERE unique1 < 10;
sum | unique1 | four
-----+---------+------
27 | 0 | 0
19 | 8 | 0
23 | 4 | 0
30 | 5 | 1
26 | 9 | 1
34 | 1 | 1
39 | 6 | 2
43 | 2 | 2
30 | 3 | 3
26 | 7 | 3
(10 rows)
SELECT sum(unique1) over (order by four groups between 2 preceding and 1 following
exclude group), unique1, four
FROM tenk1 WHERE unique1 < 10;
sum | unique1 | four
-----+---------+------
15 | 0 | 0
15 | 8 | 0
15 | 4 | 0
20 | 5 | 1
20 | 9 | 1
20 | 1 | 1
37 | 6 | 2
37 | 2 | 2
23 | 3 | 3
23 | 7 | 3
(10 rows)
SELECT sum(unique1) over (order by four groups between 2 preceding and 1 following
exclude ties), unique1, four
FROM tenk1 WHERE unique1 < 10;
sum | unique1 | four
-----+---------+------
15 | 0 | 0
23 | 8 | 0
19 | 4 | 0
25 | 5 | 1
29 | 9 | 1
21 | 1 | 1
43 | 6 | 2
39 | 2 | 2
26 | 3 | 3
30 | 7 | 3
(10 rows)
SELECT sum(unique1) over (partition by ten
order by four groups between 0 preceding and 0 following),unique1, four, ten
FROM tenk1 WHERE unique1 < 10;
sum | unique1 | four | ten
-----+---------+------+-----
0 | 0 | 0 | 0
1 | 1 | 1 | 1
2 | 2 | 2 | 2
3 | 3 | 3 | 3
4 | 4 | 0 | 4
5 | 5 | 1 | 5
6 | 6 | 2 | 6
7 | 7 | 3 | 7
8 | 8 | 0 | 8
9 | 9 | 1 | 9
(10 rows)
SELECT sum(unique1) over (partition by ten
order by four groups between 0 preceding and 0 following exclude current row), unique1, four, ten
FROM tenk1 WHERE unique1 < 10;
sum | unique1 | four | ten
-----+---------+------+-----
| 0 | 0 | 0
| 1 | 1 | 1
| 2 | 2 | 2
| 3 | 3 | 3
| 4 | 0 | 4
| 5 | 1 | 5
| 6 | 2 | 6
| 7 | 3 | 7
| 8 | 0 | 8
| 9 | 1 | 9
(10 rows)
SELECT sum(unique1) over (partition by ten
order by four groups between 0 preceding and 0 following exclude group), unique1, four, ten
FROM tenk1 WHERE unique1 < 10;
sum | unique1 | four | ten
-----+---------+------+-----
| 0 | 0 | 0
| 1 | 1 | 1
| 2 | 2 | 2
| 3 | 3 | 3
| 4 | 0 | 4
| 5 | 1 | 5
| 6 | 2 | 6
| 7 | 3 | 7
| 8 | 0 | 8
| 9 | 1 | 9
(10 rows)
SELECT sum(unique1) over (partition by ten
order by four groups between 0 preceding and 0 following exclude ties), unique1, four, ten
FROM tenk1 WHERE unique1 < 10;
sum | unique1 | four | ten
-----+---------+------+-----
0 | 0 | 0 | 0
1 | 1 | 1 | 1
2 | 2 | 2 | 2
3 | 3 | 3 | 3
4 | 4 | 0 | 4
5 | 5 | 1 | 5
6 | 6 | 2 | 6
7 | 7 | 3 | 7
8 | 8 | 0 | 8
9 | 9 | 1 | 9
(10 rows)
select first_value(salary) over(order by enroll_date groups between 1 preceding and 1 following),
lead(salary) over(order by enroll_date groups between 1 preceding and 1 following),
nth_value(salary, 1) over(order by enroll_date groups between 1 preceding and 1 following),
salary, enroll_date from empsalary;
first_value | lead | nth_value | salary | enroll_date
-------------+------+-----------+--------+-------------
5000 | 6000 | 5000 | 5000 | 10-01-2006
5000 | 3900 | 5000 | 6000 | 10-01-2006
5000 | 4800 | 5000 | 3900 | 12-23-2006
3900 | 5200 | 3900 | 4800 | 08-01-2007
3900 | 4800 | 3900 | 5200 | 08-01-2007
4800 | 5200 | 4800 | 4800 | 08-08-2007
4800 | 3500 | 4800 | 5200 | 08-15-2007
5200 | 4500 | 5200 | 3500 | 12-10-2007
3500 | 4200 | 3500 | 4500 | 01-01-2008
3500 | | 3500 | 4200 | 01-01-2008
(10 rows)
select last_value(salary) over(order by enroll_date groups between 1 preceding and 1 following),
lag(salary) over(order by enroll_date groups between 1 preceding and 1 following),
salary, enroll_date from empsalary;
last_value | lag | salary | enroll_date
------------+------+--------+-------------
3900 | | 5000 | 10-01-2006
3900 | 5000 | 6000 | 10-01-2006
5200 | 6000 | 3900 | 12-23-2006
4800 | 3900 | 4800 | 08-01-2007
4800 | 4800 | 5200 | 08-01-2007
5200 | 5200 | 4800 | 08-08-2007
3500 | 4800 | 5200 | 08-15-2007
4200 | 5200 | 3500 | 12-10-2007
4200 | 3500 | 4500 | 01-01-2008
4200 | 4500 | 4200 | 01-01-2008
(10 rows)
select first_value(salary) over(order by enroll_date groups between 1 following and 3 following
exclude current row),
lead(salary) over(order by enroll_date groups between 1 following and 3 following exclude ties),
nth_value(salary, 1) over(order by enroll_date groups between 1 following and 3 following
exclude ties),
salary, enroll_date from empsalary;
first_value | lead | nth_value | salary | enroll_date
-------------+------+-----------+--------+-------------
3900 | 6000 | 3900 | 5000 | 10-01-2006
3900 | 3900 | 3900 | 6000 | 10-01-2006
4800 | 4800 | 4800 | 3900 | 12-23-2006
4800 | 5200 | 4800 | 4800 | 08-01-2007
4800 | 4800 | 4800 | 5200 | 08-01-2007
5200 | 5200 | 5200 | 4800 | 08-08-2007
3500 | 3500 | 3500 | 5200 | 08-15-2007
4500 | 4500 | 4500 | 3500 | 12-10-2007
| 4200 | | 4500 | 01-01-2008
| | | 4200 | 01-01-2008
(10 rows)
select last_value(salary) over(order by enroll_date groups between 1 following and 3 following
exclude group),
lag(salary) over(order by enroll_date groups between 1 following and 3 following exclude group),
salary, enroll_date from empsalary;
last_value | lag | salary | enroll_date
------------+------+--------+-------------
4800 | | 5000 | 10-01-2006
4800 | 5000 | 6000 | 10-01-2006
5200 | 6000 | 3900 | 12-23-2006
3500 | 3900 | 4800 | 08-01-2007
3500 | 4800 | 5200 | 08-01-2007
4200 | 5200 | 4800 | 08-08-2007
4200 | 4800 | 5200 | 08-15-2007
4200 | 5200 | 3500 | 12-10-2007
| 3500 | 4500 | 01-01-2008
| 4500 | 4200 | 01-01-2008
(10 rows)
-- Show differences in offset interpretation between ROWS, RANGE, and GROUPS
WITH cte (x) AS (
SELECT * FROM generate_series(1, 35, 2)
)
SELECT x, (sum(x) over w)
FROM cte
WINDOW w AS (ORDER BY x rows between 1 preceding and 1 following);
x | sum
----+-----
1 | 4
3 | 9
5 | 15
7 | 21
9 | 27
11 | 33
13 | 39
15 | 45
17 | 51
19 | 57
21 | 63
23 | 69
25 | 75
27 | 81
29 | 87
31 | 93
33 | 99
35 | 68
(18 rows)
WITH cte (x) AS (
SELECT * FROM generate_series(1, 35, 2)
)
SELECT x, (sum(x) over w)
FROM cte
WINDOW w AS (ORDER BY x range between 1 preceding and 1 following);
x | sum
----+-----
1 | 1
3 | 3
5 | 5
7 | 7
9 | 9
11 | 11
13 | 13
15 | 15
17 | 17
19 | 19
21 | 21
23 | 23
25 | 25
27 | 27
29 | 29
31 | 31
33 | 33
35 | 35
(18 rows)
WITH cte (x) AS (
SELECT * FROM generate_series(1, 35, 2)
)
SELECT x, (sum(x) over w)
FROM cte
WINDOW w AS (ORDER BY x groups between 1 preceding and 1 following);
x | sum
----+-----
1 | 4
3 | 9
5 | 15
7 | 21
9 | 27
11 | 33
13 | 39
15 | 45
17 | 51
19 | 57
21 | 63
23 | 69
25 | 75
27 | 81
29 | 87
31 | 93
33 | 99
35 | 68
(18 rows)
WITH cte (x) AS (
select 1 union all select 1 union all select 1 union all
SELECT * FROM generate_series(5, 49, 2)
)
SELECT x, (sum(x) over w)
FROM cte
WINDOW w AS (ORDER BY x rows between 1 preceding and 1 following);
x | sum
----+-----
1 | 2
1 | 3
1 | 7
5 | 13
7 | 21
9 | 27
11 | 33
13 | 39
15 | 45
17 | 51
19 | 57
21 | 63
23 | 69
25 | 75
27 | 81
29 | 87
31 | 93
33 | 99
35 | 105
37 | 111
39 | 117
41 | 123
43 | 129
45 | 135
47 | 141
49 | 96
(26 rows)
WITH cte (x) AS (
select 1 union all select 1 union all select 1 union all
SELECT * FROM generate_series(5, 49, 2)
)
SELECT x, (sum(x) over w)
FROM cte
WINDOW w AS (ORDER BY x range between 1 preceding and 1 following);
x | sum
----+-----
1 | 3
1 | 3
1 | 3
5 | 5
7 | 7
9 | 9
11 | 11
13 | 13
15 | 15
17 | 17
19 | 19
21 | 21
23 | 23
25 | 25
27 | 27
29 | 29
31 | 31
33 | 33
35 | 35
37 | 37
39 | 39
41 | 41
43 | 43
45 | 45
47 | 47
49 | 49
(26 rows)
WITH cte (x) AS (
select 1 union all select 1 union all select 1 union all
SELECT * FROM generate_series(5, 49, 2)
)
SELECT x, (sum(x) over w)
FROM cte
WINDOW w AS (ORDER BY x groups between 1 preceding and 1 following);
x | sum
----+-----
1 | 8
1 | 8
1 | 8
5 | 15
7 | 21
9 | 27
11 | 33
13 | 39
15 | 45
17 | 51
19 | 57
21 | 63
23 | 69
25 | 75
27 | 81
29 | 87
31 | 93
33 | 99
35 | 105
37 | 111
39 | 117
41 | 123
43 | 129
45 | 135
47 | 141
49 | 96
(26 rows)
-- with UNION
SELECT count(*) OVER (PARTITION BY four) FROM (SELECT * FROM tenk1 UNION ALL SELECT * FROM tenk2)s LIMIT 0;
count
-------
(0 rows)
-- check some degenerate cases
create temp table t1 (f1 int, f2 int8);
insert into t1 values (1,1),(1,2),(2,2);
select f1, sum(f1) over (partition by f1
range between 1 preceding and 1 following)
from t1 where f1 = f2; -- error, must have order by
ERROR: RANGE with offset PRECEDING/FOLLOWING requires exactly one ORDER BY column
LINE 1: select f1, sum(f1) over (partition by f1
^
explain (costs off)
select f1, sum(f1) over (partition by f1 order by f2
range between 1 preceding and 1 following)
from t1 where f1 = f2;
QUERY PLAN
---------------------------------
WindowAgg
-> Sort
Sort Key: f1
-> Seq Scan on t1
Filter: (f1 = f2)
(5 rows)
select f1, sum(f1) over (partition by f1 order by f2
range between 1 preceding and 1 following)
from t1 where f1 = f2;
f1 | sum
----+-----
1 | 1
2 | 2
(2 rows)
select f1, sum(f1) over (partition by f1, f1 order by f2
range between 2 preceding and 1 preceding)
from t1 where f1 = f2;
f1 | sum
----+-----
1 |
2 |
(2 rows)
select f1, sum(f1) over (partition by f1, f2 order by f2
range between 1 following and 2 following)
from t1 where f1 = f2;
f1 | sum
----+-----
1 |
2 |
(2 rows)
select f1, sum(f1) over (partition by f1
groups between 1 preceding and 1 following)
from t1 where f1 = f2; -- error, must have order by
ERROR: GROUPS mode requires an ORDER BY clause
LINE 1: select f1, sum(f1) over (partition by f1
^
explain (costs off)
select f1, sum(f1) over (partition by f1 order by f2
groups between 1 preceding and 1 following)
from t1 where f1 = f2;
QUERY PLAN
---------------------------------
WindowAgg
-> Sort
Sort Key: f1
-> Seq Scan on t1
Filter: (f1 = f2)
(5 rows)
select f1, sum(f1) over (partition by f1 order by f2
groups between 1 preceding and 1 following)
from t1 where f1 = f2;
f1 | sum
----+-----
1 | 1
2 | 2
(2 rows)
select f1, sum(f1) over (partition by f1, f1 order by f2
groups between 2 preceding and 1 preceding)
from t1 where f1 = f2;
f1 | sum
----+-----
1 |
2 |
(2 rows)
select f1, sum(f1) over (partition by f1, f2 order by f2
groups between 1 following and 2 following)
from t1 where f1 = f2;
f1 | sum
----+-----
1 |
2 |
(2 rows)
-- ordering by a non-integer constant is allowed
SELECT rank() OVER (ORDER BY length('abc'));
rank
------
1
(1 row)
-- can't order by another window function
SELECT rank() OVER (ORDER BY rank() OVER (ORDER BY random()));
ERROR: window functions are not allowed in window definitions
LINE 1: SELECT rank() OVER (ORDER BY rank() OVER (ORDER BY random())...
^
-- some other errors
SELECT * FROM empsalary WHERE row_number() OVER (ORDER BY salary) < 10;
ERROR: window functions are not allowed in WHERE
LINE 1: SELECT * FROM empsalary WHERE row_number() OVER (ORDER BY sa...
^
SELECT * FROM empsalary INNER JOIN tenk1 ON row_number() OVER (ORDER BY salary) < 10;
ERROR: window functions are not allowed in JOIN conditions
LINE 1: SELECT * FROM empsalary INNER JOIN tenk1 ON row_number() OVE...
^
SELECT rank() OVER (ORDER BY 1), count(*) FROM empsalary GROUP BY 1;
ERROR: window functions are not allowed in GROUP BY
LINE 1: SELECT rank() OVER (ORDER BY 1), count(*) FROM empsalary GRO...
^
SELECT * FROM rank() OVER (ORDER BY random());
ERROR: syntax error at or near "ORDER"
LINE 1: SELECT * FROM rank() OVER (ORDER BY random());
^
DELETE FROM empsalary WHERE (rank() OVER (ORDER BY random())) > 10;
ERROR: window functions are not allowed in WHERE
LINE 1: DELETE FROM empsalary WHERE (rank() OVER (ORDER BY random())...
^
DELETE FROM empsalary RETURNING rank() OVER (ORDER BY random());
ERROR: window functions are not allowed in RETURNING
LINE 1: DELETE FROM empsalary RETURNING rank() OVER (ORDER BY random...
^
SELECT count(*) OVER w FROM tenk1 WINDOW w AS (ORDER BY unique1), w AS (ORDER BY unique1);
ERROR: window "w" is already defined
LINE 1: ...w FROM tenk1 WINDOW w AS (ORDER BY unique1), w AS (ORDER BY ...
^
SELECT rank() OVER (PARTITION BY four, ORDER BY ten) FROM tenk1;
ERROR: syntax error at or near "ORDER"
LINE 1: SELECT rank() OVER (PARTITION BY four, ORDER BY ten) FROM te...
^
SELECT count() OVER () FROM tenk1;
ERROR: count(*) must be used to call a parameterless aggregate function
LINE 1: SELECT count() OVER () FROM tenk1;
^
SELECT generate_series(1, 100) OVER () FROM empsalary;
ERROR: OVER specified, but generate_series is not a window function nor an aggregate function
LINE 1: SELECT generate_series(1, 100) OVER () FROM empsalary;
^
SELECT ntile(0) OVER (ORDER BY ten), ten, four FROM tenk1;
ERROR: argument of ntile must be greater than zero
SELECT nth_value(four, 0) OVER (ORDER BY ten), ten, four FROM tenk1;
ERROR: argument of nth_value must be greater than zero
-- filter
SELECT sum(salary), row_number() OVER (ORDER BY depname), sum(
sum(salary) FILTER (WHERE enroll_date > '2007-01-01')
) FILTER (WHERE depname <> 'sales') OVER (ORDER BY depname DESC) AS "filtered_sum",
depname
FROM empsalary GROUP BY depname;
sum | row_number | filtered_sum | depname
-------+------------+--------------+-----------
14600 | 3 | | sales
7400 | 2 | 3500 | personnel
25100 | 1 | 22600 | develop
(3 rows)
-- Test pushdown of quals into a subquery containing window functions
-- pushdown is safe because all PARTITION BY clauses include depname:
EXPLAIN (COSTS OFF)
SELECT * FROM
(SELECT depname,
sum(salary) OVER (PARTITION BY depname) depsalary,
min(salary) OVER (PARTITION BY depname || 'A', depname) depminsalary
FROM empsalary) emp
WHERE depname = 'sales';
QUERY PLAN
---------------------------------------------------------------------
Subquery Scan on emp
-> WindowAgg
-> Sort
Sort Key: (((empsalary.depname)::text || 'A'::text))
-> WindowAgg
-> Seq Scan on empsalary
Filter: ((depname)::text = 'sales'::text)
(7 rows)
-- pushdown is unsafe because there's a PARTITION BY clause without depname:
EXPLAIN (COSTS OFF)
SELECT * FROM
(SELECT depname,
sum(salary) OVER (PARTITION BY enroll_date) enroll_salary,
min(salary) OVER (PARTITION BY depname) depminsalary
FROM empsalary) emp
WHERE depname = 'sales';
QUERY PLAN
-----------------------------------------------------------
Subquery Scan on emp
Filter: ((emp.depname)::text = 'sales'::text)
-> WindowAgg
-> Sort
Sort Key: empsalary.depname
-> WindowAgg
-> Sort
Sort Key: empsalary.enroll_date
-> Seq Scan on empsalary
(9 rows)
-- cleanup
DROP TABLE empsalary;
-- test user-defined window function with named args and default args
CREATE FUNCTION nth_value_def(val anyelement, n integer = 1) RETURNS anyelement
LANGUAGE internal WINDOW IMMUTABLE STRICT AS 'window_nth_value';
SELECT nth_value_def(n := 2, val := ten) OVER (PARTITION BY four), ten, four
FROM (SELECT * FROM tenk1 WHERE unique2 < 10 ORDER BY four, ten) s;
nth_value_def | ten | four
---------------+-----+------
0 | 0 | 0
0 | 0 | 0
0 | 4 | 0
1 | 1 | 1
1 | 1 | 1
1 | 7 | 1
1 | 9 | 1
| 0 | 2
3 | 1 | 3
3 | 3 | 3
(10 rows)
SELECT nth_value_def(ten) OVER (PARTITION BY four), ten, four
FROM (SELECT * FROM tenk1 WHERE unique2 < 10 ORDER BY four, ten) s;
nth_value_def | ten | four
---------------+-----+------
0 | 0 | 0
0 | 0 | 0
0 | 4 | 0
1 | 1 | 1
1 | 1 | 1
1 | 7 | 1
1 | 9 | 1
0 | 0 | 2
1 | 1 | 3
1 | 3 | 3
(10 rows)
--
-- Test the basic moving-aggregate machinery
--
-- create aggregates that record the series of transform calls (these are
-- intentionally not true inverses)
CREATE FUNCTION logging_sfunc_nonstrict(text, anyelement) RETURNS text AS
$$ SELECT COALESCE($1, '') || '*' || quote_nullable($2) $$
LANGUAGE SQL IMMUTABLE;
CREATE FUNCTION logging_msfunc_nonstrict(text, anyelement) RETURNS text AS
$$ SELECT COALESCE($1, '') || '+' || quote_nullable($2) $$
LANGUAGE SQL IMMUTABLE;
CREATE FUNCTION logging_minvfunc_nonstrict(text, anyelement) RETURNS text AS
$$ SELECT $1 || '-' || quote_nullable($2) $$
LANGUAGE SQL IMMUTABLE;
CREATE AGGREGATE logging_agg_nonstrict (anyelement)
(
stype = text,
sfunc = logging_sfunc_nonstrict,
mstype = text,
msfunc = logging_msfunc_nonstrict,
minvfunc = logging_minvfunc_nonstrict
);
CREATE AGGREGATE logging_agg_nonstrict_initcond (anyelement)
(
stype = text,
sfunc = logging_sfunc_nonstrict,
mstype = text,
msfunc = logging_msfunc_nonstrict,
minvfunc = logging_minvfunc_nonstrict,
initcond = 'I',
minitcond = 'MI'
);
CREATE FUNCTION logging_sfunc_strict(text, anyelement) RETURNS text AS
$$ SELECT $1 || '*' || quote_nullable($2) $$
LANGUAGE SQL STRICT IMMUTABLE;
CREATE FUNCTION logging_msfunc_strict(text, anyelement) RETURNS text AS
$$ SELECT $1 || '+' || quote_nullable($2) $$
LANGUAGE SQL STRICT IMMUTABLE;
CREATE FUNCTION logging_minvfunc_strict(text, anyelement) RETURNS text AS
$$ SELECT $1 || '-' || quote_nullable($2) $$
LANGUAGE SQL STRICT IMMUTABLE;
CREATE AGGREGATE logging_agg_strict (text)
(
stype = text,
sfunc = logging_sfunc_strict,
mstype = text,
msfunc = logging_msfunc_strict,
minvfunc = logging_minvfunc_strict
);
CREATE AGGREGATE logging_agg_strict_initcond (anyelement)
(
stype = text,
sfunc = logging_sfunc_strict,
mstype = text,
msfunc = logging_msfunc_strict,
minvfunc = logging_minvfunc_strict,
initcond = 'I',
minitcond = 'MI'
);
-- test strict and non-strict cases
SELECT
p::text || ',' || i::text || ':' || COALESCE(v::text, 'NULL') AS row,
logging_agg_nonstrict(v) over wnd as nstrict,
logging_agg_nonstrict_initcond(v) over wnd as nstrict_init,
logging_agg_strict(v::text) over wnd as strict,
logging_agg_strict_initcond(v) over wnd as strict_init
FROM (VALUES
(1, 1, NULL),
(1, 2, 'a'),
(1, 3, 'b'),
(1, 4, NULL),
(1, 5, NULL),
(1, 6, 'c'),
(2, 1, NULL),
(2, 2, 'x'),
(3, 1, 'z')
) AS t(p, i, v)
WINDOW wnd AS (PARTITION BY P ORDER BY i ROWS BETWEEN 1 PRECEDING AND CURRENT ROW)
ORDER BY p, i;
row | nstrict | nstrict_init | strict | strict_init
----------+-----------------------------------------------+-------------------------------------------------+-----------+----------------
1,1:NULL | +NULL | MI+NULL | | MI
1,2:a | +NULL+'a' | MI+NULL+'a' | a | MI+'a'
1,3:b | +NULL+'a'-NULL+'b' | MI+NULL+'a'-NULL+'b' | a+'b' | MI+'a'+'b'
1,4:NULL | +NULL+'a'-NULL+'b'-'a'+NULL | MI+NULL+'a'-NULL+'b'-'a'+NULL | a+'b'-'a' | MI+'a'+'b'-'a'
1,5:NULL | +NULL+'a'-NULL+'b'-'a'+NULL-'b'+NULL | MI+NULL+'a'-NULL+'b'-'a'+NULL-'b'+NULL | | MI
1,6:c | +NULL+'a'-NULL+'b'-'a'+NULL-'b'+NULL-NULL+'c' | MI+NULL+'a'-NULL+'b'-'a'+NULL-'b'+NULL-NULL+'c' | c | MI+'c'
2,1:NULL | +NULL | MI+NULL | | MI
2,2:x | +NULL+'x' | MI+NULL+'x' | x | MI+'x'
3,1:z | +'z' | MI+'z' | z | MI+'z'
(9 rows)
-- and again, but with filter
SELECT
p::text || ',' || i::text || ':' ||
CASE WHEN f THEN COALESCE(v::text, 'NULL') ELSE '-' END as row,
logging_agg_nonstrict(v) filter(where f) over wnd as nstrict_filt,
logging_agg_nonstrict_initcond(v) filter(where f) over wnd as nstrict_init_filt,
logging_agg_strict(v::text) filter(where f) over wnd as strict_filt,
logging_agg_strict_initcond(v) filter(where f) over wnd as strict_init_filt
FROM (VALUES
(1, 1, true, NULL),
(1, 2, false, 'a'),
(1, 3, true, 'b'),
(1, 4, false, NULL),
(1, 5, false, NULL),
(1, 6, false, 'c'),
(2, 1, false, NULL),
(2, 2, true, 'x'),
(3, 1, true, 'z')
) AS t(p, i, f, v)
WINDOW wnd AS (PARTITION BY p ORDER BY i ROWS BETWEEN 1 PRECEDING AND CURRENT ROW)
ORDER BY p, i;
row | nstrict_filt | nstrict_init_filt | strict_filt | strict_init_filt
----------+--------------+-------------------+-------------+------------------
1,1:NULL | +NULL | MI+NULL | | MI
1,2:- | +NULL | MI+NULL | | MI
1,3:b | +'b' | MI+'b' | b | MI+'b'
1,4:- | +'b' | MI+'b' | b | MI+'b'
1,5:- | | MI | | MI
1,6:- | | MI | | MI
2,1:- | | MI | | MI
2,2:x | +'x' | MI+'x' | x | MI+'x'
3,1:z | +'z' | MI+'z' | z | MI+'z'
(9 rows)
-- test that volatile arguments disable moving-aggregate mode
SELECT
i::text || ':' || COALESCE(v::text, 'NULL') as row,
logging_agg_strict(v::text)
over wnd as inverse,
logging_agg_strict(v::text || CASE WHEN random() < 0 then '?' ELSE '' END)
over wnd as noinverse
FROM (VALUES
(1, 'a'),
(2, 'b'),
(3, 'c')
) AS t(i, v)
WINDOW wnd AS (ORDER BY i ROWS BETWEEN 1 PRECEDING AND CURRENT ROW)
ORDER BY i;
row | inverse | noinverse
-----+---------------+-----------
1:a | a | a
2:b | a+'b' | a*'b'
3:c | a+'b'-'a'+'c' | b*'c'
(3 rows)
SELECT
i::text || ':' || COALESCE(v::text, 'NULL') as row,
logging_agg_strict(v::text) filter(where true)
over wnd as inverse,
logging_agg_strict(v::text) filter(where random() >= 0)
over wnd as noinverse
FROM (VALUES
(1, 'a'),
(2, 'b'),
(3, 'c')
) AS t(i, v)
WINDOW wnd AS (ORDER BY i ROWS BETWEEN 1 PRECEDING AND CURRENT ROW)
ORDER BY i;
row | inverse | noinverse
-----+---------------+-----------
1:a | a | a
2:b | a+'b' | a*'b'
3:c | a+'b'-'a'+'c' | b*'c'
(3 rows)
-- test that non-overlapping windows don't use inverse transitions
SELECT
logging_agg_strict(v::text) OVER wnd
FROM (VALUES
(1, 'a'),
(2, 'b'),
(3, 'c')
) AS t(i, v)
WINDOW wnd AS (ORDER BY i ROWS BETWEEN CURRENT ROW AND CURRENT ROW)
ORDER BY i;
logging_agg_strict
--------------------
a
b
c
(3 rows)
-- test that returning NULL from the inverse transition functions
-- restarts the aggregation from scratch. The second aggregate is supposed
-- to test cases where only some aggregates restart, the third one checks
-- that one aggregate restarting doesn't cause others to restart.
CREATE FUNCTION sum_int_randrestart_minvfunc(int4, int4) RETURNS int4 AS
$$ SELECT CASE WHEN random() < 0.2 THEN NULL ELSE $1 - $2 END $$
LANGUAGE SQL STRICT;
CREATE AGGREGATE sum_int_randomrestart (int4)
(
stype = int4,
sfunc = int4pl,
mstype = int4,
msfunc = int4pl,
minvfunc = sum_int_randrestart_minvfunc
);
WITH
vs AS (
SELECT i, (random() * 100)::int4 AS v
FROM generate_series(1, 100) AS i
),
sum_following AS (
SELECT i, SUM(v) OVER
(ORDER BY i DESC ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS s
FROM vs
)
SELECT DISTINCT
sum_following.s = sum_int_randomrestart(v) OVER fwd AS eq1,
-sum_following.s = sum_int_randomrestart(-v) OVER fwd AS eq2,
100*3+(vs.i-1)*3 = length(logging_agg_nonstrict(''::text) OVER fwd) AS eq3
FROM vs
JOIN sum_following ON sum_following.i = vs.i
WINDOW fwd AS (
ORDER BY vs.i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
);
eq1 | eq2 | eq3
-----+-----+-----
t | t | t
(1 row)
--
-- Test various built-in aggregates that have moving-aggregate support
--
-- test inverse transition functions handle NULLs properly
SELECT i,AVG(v::bigint) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,1),(2,2),(3,NULL),(4,NULL)) t(i,v);
i | avg
---+--------------------
1 | 1.5000000000000000
2 | 2.0000000000000000
3 |
4 |
(4 rows)
SELECT i,AVG(v::int) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,1),(2,2),(3,NULL),(4,NULL)) t(i,v);
i | avg
---+--------------------
1 | 1.5000000000000000
2 | 2.0000000000000000
3 |
4 |
(4 rows)
SELECT i,AVG(v::smallint) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,1),(2,2),(3,NULL),(4,NULL)) t(i,v);
i | avg
---+--------------------
1 | 1.5000000000000000
2 | 2.0000000000000000
3 |
4 |
(4 rows)
SELECT i,AVG(v::numeric) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,1.5),(2,2.5),(3,NULL),(4,NULL)) t(i,v);
i | avg
---+--------------------
1 | 2.0000000000000000
2 | 2.5000000000000000
3 |
4 |
(4 rows)
SELECT i,AVG(v::interval) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,'1 sec'),(2,'2 sec'),(3,NULL),(4,NULL)) t(i,v);
i | avg
---+------------
1 | @ 1.5 secs
2 | @ 2 secs
3 |
4 |
(4 rows)
SELECT i,SUM(v::smallint) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,1),(2,2),(3,NULL),(4,NULL)) t(i,v);
i | sum
---+-----
1 | 3
2 | 2
3 |
4 |
(4 rows)
SELECT i,SUM(v::int) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,1),(2,2),(3,NULL),(4,NULL)) t(i,v);
i | sum
---+-----
1 | 3
2 | 2
3 |
4 |
(4 rows)
SELECT i,SUM(v::bigint) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,1),(2,2),(3,NULL),(4,NULL)) t(i,v);
i | sum
---+-----
1 | 3
2 | 2
3 |
4 |
(4 rows)
SELECT i,SUM(v::money) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,'1.10'),(2,'2.20'),(3,NULL),(4,NULL)) t(i,v);
i | sum
---+-------
1 | $3.30
2 | $2.20
3 |
4 |
(4 rows)
SELECT i,SUM(v::interval) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,'1 sec'),(2,'2 sec'),(3,NULL),(4,NULL)) t(i,v);
i | sum
---+----------
1 | @ 3 secs
2 | @ 2 secs
3 |
4 |
(4 rows)
SELECT i,SUM(v::numeric) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,1.1),(2,2.2),(3,NULL),(4,NULL)) t(i,v);
i | sum
---+-----
1 | 3.3
2 | 2.2
3 |
4 |
(4 rows)
SELECT SUM(n::numeric) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,1.01),(2,2),(3,3)) v(i,n);
sum
------
6.01
5
3
(3 rows)
SELECT i,COUNT(v) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,1),(2,2),(3,NULL),(4,NULL)) t(i,v);
i | count
---+-------
1 | 2
2 | 1
3 | 0
4 | 0
(4 rows)
SELECT i,COUNT(*) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,1),(2,2),(3,NULL),(4,NULL)) t(i,v);
i | count
---+-------
1 | 4
2 | 3
3 | 2
4 | 1
(4 rows)
SELECT VAR_POP(n::bigint) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,600),(2,470),(3,170),(4,430),(5,300)) r(i,n);
var_pop
-----------------------
21704.000000000000
13868.750000000000
11266.666666666667
4225.0000000000000000
0
(5 rows)
SELECT VAR_POP(n::int) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,600),(2,470),(3,170),(4,430),(5,300)) r(i,n);
var_pop
-----------------------
21704.000000000000
13868.750000000000
11266.666666666667
4225.0000000000000000
0
(5 rows)
SELECT VAR_POP(n::smallint) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,600),(2,470),(3,170),(4,430),(5,300)) r(i,n);
var_pop
-----------------------
21704.000000000000
13868.750000000000
11266.666666666667
4225.0000000000000000
0
(5 rows)
SELECT VAR_POP(n::numeric) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,600),(2,470),(3,170),(4,430),(5,300)) r(i,n);
var_pop
-----------------------
21704.000000000000
13868.750000000000
11266.666666666667
4225.0000000000000000
0
(5 rows)
SELECT VAR_SAMP(n::bigint) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,600),(2,470),(3,170),(4,430),(5,300)) r(i,n);
var_samp
-----------------------
27130.000000000000
18491.666666666667
16900.000000000000
8450.0000000000000000
(5 rows)
SELECT VAR_SAMP(n::int) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,600),(2,470),(3,170),(4,430),(5,300)) r(i,n);
var_samp
-----------------------
27130.000000000000
18491.666666666667
16900.000000000000
8450.0000000000000000
(5 rows)
SELECT VAR_SAMP(n::smallint) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,600),(2,470),(3,170),(4,430),(5,300)) r(i,n);
var_samp
-----------------------
27130.000000000000
18491.666666666667
16900.000000000000
8450.0000000000000000
(5 rows)
SELECT VAR_SAMP(n::numeric) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,600),(2,470),(3,170),(4,430),(5,300)) r(i,n);
var_samp
-----------------------
27130.000000000000
18491.666666666667
16900.000000000000
8450.0000000000000000
(5 rows)
SELECT VARIANCE(n::bigint) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,600),(2,470),(3,170),(4,430),(5,300)) r(i,n);
variance
-----------------------
27130.000000000000
18491.666666666667
16900.000000000000
8450.0000000000000000
(5 rows)
SELECT VARIANCE(n::int) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,600),(2,470),(3,170),(4,430),(5,300)) r(i,n);
variance
-----------------------
27130.000000000000
18491.666666666667
16900.000000000000
8450.0000000000000000
(5 rows)
SELECT VARIANCE(n::smallint) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,600),(2,470),(3,170),(4,430),(5,300)) r(i,n);
variance
-----------------------
27130.000000000000
18491.666666666667
16900.000000000000
8450.0000000000000000
(5 rows)
SELECT VARIANCE(n::numeric) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,600),(2,470),(3,170),(4,430),(5,300)) r(i,n);
variance
-----------------------
27130.000000000000
18491.666666666667
16900.000000000000
8450.0000000000000000
(5 rows)
SELECT STDDEV_POP(n::bigint) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,NULL),(2,600),(3,470),(4,170),(5,430),(6,300)) r(i,n);
stddev_pop
---------------------
147.322774885623
147.322774885623
117.765657133139
106.144555520604
65.0000000000000000
0
(6 rows)
SELECT STDDEV_POP(n::int) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,NULL),(2,600),(3,470),(4,170),(5,430),(6,300)) r(i,n);
stddev_pop
---------------------
147.322774885623
147.322774885623
117.765657133139
106.144555520604
65.0000000000000000
0
(6 rows)
SELECT STDDEV_POP(n::smallint) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,NULL),(2,600),(3,470),(4,170),(5,430),(6,300)) r(i,n);
stddev_pop
---------------------
147.322774885623
147.322774885623
117.765657133139
106.144555520604
65.0000000000000000
0
(6 rows)
SELECT STDDEV_POP(n::numeric) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,NULL),(2,600),(3,470),(4,170),(5,430),(6,300)) r(i,n);
stddev_pop
---------------------
147.322774885623
147.322774885623
117.765657133139
106.144555520604
65.0000000000000000
0
(6 rows)
SELECT STDDEV_SAMP(n::bigint) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,NULL),(2,600),(3,470),(4,170),(5,430),(6,300)) r(i,n);
stddev_samp
---------------------
164.711869639076
164.711869639076
135.984067694222
130.000000000000
91.9238815542511782
(6 rows)
SELECT STDDEV_SAMP(n::int) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,NULL),(2,600),(3,470),(4,170),(5,430),(6,300)) r(i,n);
stddev_samp
---------------------
164.711869639076
164.711869639076
135.984067694222
130.000000000000
91.9238815542511782
(6 rows)
SELECT STDDEV_SAMP(n::smallint) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,NULL),(2,600),(3,470),(4,170),(5,430),(6,300)) r(i,n);
stddev_samp
---------------------
164.711869639076
164.711869639076
135.984067694222
130.000000000000
91.9238815542511782
(6 rows)
SELECT STDDEV_SAMP(n::numeric) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(1,NULL),(2,600),(3,470),(4,170),(5,430),(6,300)) r(i,n);
stddev_samp
---------------------
164.711869639076
164.711869639076
135.984067694222
130.000000000000
91.9238815542511782
(6 rows)
SELECT STDDEV(n::bigint) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(0,NULL),(1,600),(2,470),(3,170),(4,430),(5,300)) r(i,n);
stddev
---------------------
164.711869639076
164.711869639076
135.984067694222
130.000000000000
91.9238815542511782
(6 rows)
SELECT STDDEV(n::int) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(0,NULL),(1,600),(2,470),(3,170),(4,430),(5,300)) r(i,n);
stddev
---------------------
164.711869639076
164.711869639076
135.984067694222
130.000000000000
91.9238815542511782
(6 rows)
SELECT STDDEV(n::smallint) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(0,NULL),(1,600),(2,470),(3,170),(4,430),(5,300)) r(i,n);
stddev
---------------------
164.711869639076
164.711869639076
135.984067694222
130.000000000000
91.9238815542511782
(6 rows)
SELECT STDDEV(n::numeric) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
FROM (VALUES(0,NULL),(1,600),(2,470),(3,170),(4,430),(5,300)) r(i,n);
stddev
---------------------
164.711869639076
164.711869639076
135.984067694222
130.000000000000
91.9238815542511782
(6 rows)
-- test that inverse transition functions work with various frame options
SELECT i,SUM(v::int) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND CURRENT ROW)
FROM (VALUES(1,1),(2,2),(3,NULL),(4,NULL)) t(i,v);
i | sum
---+-----
1 | 1
2 | 2
3 |
4 |
(4 rows)
SELECT i,SUM(v::int) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND 1 FOLLOWING)
FROM (VALUES(1,1),(2,2),(3,NULL),(4,NULL)) t(i,v);
i | sum
---+-----
1 | 3
2 | 2
3 |
4 |
(4 rows)
SELECT i,SUM(v::int) OVER (ORDER BY i ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING)
FROM (VALUES(1,1),(2,2),(3,3),(4,4)) t(i,v);
i | sum
---+-----
1 | 3
2 | 6
3 | 9
4 | 7
(4 rows)
-- ensure aggregate over numeric properly recovers from NaN values
SELECT a, b,
SUM(b) OVER(ORDER BY A ROWS BETWEEN 1 PRECEDING AND CURRENT ROW)
FROM (VALUES(1,1::numeric),(2,2),(3,'NaN'),(4,3),(5,4)) t(a,b);
a | b | sum
---+-----+-----
1 | 1 | 1
2 | 2 | 3
3 | NaN | NaN
4 | 3 | NaN
5 | 4 | 7
(5 rows)
-- It might be tempting for someone to add an inverse trans function for
-- float and double precision. This should not be done as it can give incorrect
-- results. This test should fail if anyone ever does this without thinking too
-- hard about it.
SELECT to_char(SUM(n::float8) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND 1 FOLLOWING),'999999999999999999999D9')
FROM (VALUES(1,1e20),(2,1)) n(i,n);
to_char
--------------------------
100000000000000000000
1.0
(2 rows)
SELECT i, b, bool_and(b) OVER w, bool_or(b) OVER w
FROM (VALUES (1,true), (2,true), (3,false), (4,false), (5,true)) v(i,b)
WINDOW w AS (ORDER BY i ROWS BETWEEN CURRENT ROW AND 1 FOLLOWING);
i | b | bool_and | bool_or
---+---+----------+---------
1 | t | t | t
2 | t | f | t
3 | f | f | f
4 | f | f | t
5 | t | t | t
(5 rows)
Attachments:
[text/x-diff] 0001-Inlining-CTEs-v0002.patch (15.9K, ../../20180725054236.GB17411@fetter.org/2-0001-Inlining-CTEs-v0002.patch)
download | inline diff:
From 2838de4fe03cce59c7e4c0d2dc6b09cba4cad11e Mon Sep 17 00:00:00 2001
From: David Fetter <david@fetter.org>
Date: Tue, 24 Jul 2018 22:09:55 -0700
Subject: [PATCH] Inlining CTEs v0002
To: pgsql-hackers@postgresql.org
---
src/backend/nodes/copyfuncs.c | 1 +
src/backend/nodes/equalfuncs.c | 1 +
src/backend/nodes/outfuncs.c | 1 +
src/backend/nodes/readfuncs.c | 1 +
src/backend/optimizer/plan/subselect.c | 152 ++++++++++++++++++++++
src/include/nodes/parsenodes.h | 1 +
src/test/regress/expected/rowsecurity.out | 92 ++++++-------
src/test/regress/expected/rowtypes.out | 13 +-
src/test/regress/expected/rules.out | 6 +-
9 files changed, 201 insertions(+), 67 deletions(-)
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 17b650b8cb..8ba2fc1aed 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2530,6 +2530,7 @@ _copyCommonTableExpr(const CommonTableExpr *from)
COPY_NODE_FIELD(ctequery);
COPY_LOCATION_FIELD(location);
COPY_SCALAR_FIELD(cterecursive);
+ COPY_SCALAR_FIELD(ctematerialized);
COPY_SCALAR_FIELD(cterefcount);
COPY_NODE_FIELD(ctecolnames);
COPY_NODE_FIELD(ctecoltypes);
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index 378f2facb8..2a04506d0d 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2793,6 +2793,7 @@ _equalCommonTableExpr(const CommonTableExpr *a, const CommonTableExpr *b)
COMPARE_NODE_FIELD(ctequery);
COMPARE_LOCATION_FIELD(location);
COMPARE_SCALAR_FIELD(cterecursive);
+ COMPARE_SCALAR_FIELD(ctematerialized);
COMPARE_SCALAR_FIELD(cterefcount);
COMPARE_NODE_FIELD(ctecolnames);
COMPARE_NODE_FIELD(ctecoltypes);
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index a6454ce28b..0ea6b7125a 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -3080,6 +3080,7 @@ _outCommonTableExpr(StringInfo str, const CommonTableExpr *node)
WRITE_NODE_FIELD(ctequery);
WRITE_LOCATION_FIELD(location);
WRITE_BOOL_FIELD(cterecursive);
+ WRITE_BOOL_FIELD(ctematerialized);
WRITE_INT_FIELD(cterefcount);
WRITE_NODE_FIELD(ctecolnames);
WRITE_NODE_FIELD(ctecoltypes);
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index 9a01eb6b63..01db3c1c85 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -409,6 +409,7 @@ _readCommonTableExpr(void)
READ_NODE_FIELD(ctequery);
READ_LOCATION_FIELD(location);
READ_BOOL_FIELD(cterecursive);
+ READ_BOOL_FIELD(ctematerialized);
READ_INT_FIELD(cterefcount);
READ_NODE_FIELD(ctecolnames);
READ_NODE_FIELD(ctecoltypes);
diff --git a/src/backend/optimizer/plan/subselect.c b/src/backend/optimizer/plan/subselect.c
index 83008d7661..e4d71f4652 100644
--- a/src/backend/optimizer/plan/subselect.c
+++ b/src/backend/optimizer/plan/subselect.c
@@ -1146,6 +1146,141 @@ hash_ok_operator(OpExpr *expr)
}
+struct inline_cte_walker_ctx
+{
+ const char *ctename;
+ int levelsup;
+ int refcount;
+ Query *ctequery;
+ CommonTableExpr *cte;
+};
+
+static bool inline_cte_walker(Node *node, void *ctxp)
+{
+ struct inline_cte_walker_ctx *ctx = ctxp;
+
+ if (!node)
+ return false;
+
+ if (IsA(node, Query))
+ {
+ /*
+ * This one is a bit tricky. It's our job to handle the recursion here,
+ * but we do some of the lifting normally handled by query_tree_walker
+ * in order to get the sequence of operations right.
+ *
+ * First, if the Query we're looking at is the one containing our CTE
+ * definition, then we don't need to recurse into our own CTE or CTEs
+ * that are earlier in the list than ours (since cteList has been
+ * sorted for us into dependency order). We could check whether a
+ * nested query is hiding ours, but that seems too much of an edge case
+ * to be worth optimizing (the levelsup check will ensure we don't
+ * replace any CTEs improperly). So we scan the cteList ourselves
+ * rather than having query_tree_walker do it.
+ *
+ * Second, we want to walk the rangetable _before_ replacing any
+ * RTE_CTE nodes, in order to avoid re-walking the subquery we just
+ * inlined. (range_table_walker, if told to visit the RTE nodes at all,
+ * visits them before their content.) So we have range_table_walker
+ * ignore the RTE nodes themselves and only walk their contents.
+ *
+ * Third, we scan the rangetable for RTE_CTE nodes to replace.
+ *
+ * Fourth, we use query_tree_walker to find and walk the rest of the
+ * query, telling it to ignore the rangetable and CTEs.
+ *
+ * Note that ctx->levelsup is -1 on entry the first time, since we need
+ * the incremented value to be 0 when scanning the content of the query
+ * containing the definition.
+ */
+ Query *query = castNode(Query, node);
+ ListCell *lc;
+ bool do_replace = ctx->levelsup >= 0;
+
+ ctx->levelsup++;
+
+ foreach (lc, query->cteList)
+ {
+ CommonTableExpr *cte = lfirst_node(CommonTableExpr, lc);
+
+ if (!do_replace && strcmp(cte->ctename, ctx->ctename) == 0)
+ do_replace = true;
+ else if (do_replace)
+ inline_cte_walker(cte->ctequery, ctxp);
+ }
+
+ range_table_walker(query->rtable, inline_cte_walker, ctxp, 0);
+
+ foreach (lc, query->rtable)
+ {
+ RangeTblEntry *rte = lfirst_node(RangeTblEntry, lc);
+
+ if (rte->rtekind == RTE_CTE &&
+ strcmp(rte->ctename, ctx->ctename) == 0 &&
+ rte->ctelevelsup == ctx->levelsup)
+ {
+ Query *newquery = ctx->ctequery;
+
+ /*
+ * We need to do some work here that view rewrite does not, and
+ * in turn we do not do some work that view rewrite does.
+ *
+ * Firstly, views can't have outer references but CTEs can
+ * (especially in the case of CTEs referencing other CTEs), so
+ * we need to fix up all levelsup attributes inside the CTE
+ * query.
+ *
+ * Secondly, views (and explicit subqueries) currently have
+ * different behaviour w.r.t. SELECT FOR UPDATE than CTEs do. A
+ * FOR UPDATE clause is treated as extending into views and
+ * subqueries, but not into CTEs. We preserve this distinction
+ * by not trying to push rowmarks into the new subquery.
+ *
+ * We avoid copyObject if possible because subquery processing
+ * copies the query too.
+ */
+ if (ctx->levelsup > 0)
+ {
+ newquery = copyObject(newquery);
+ IncrementVarSublevelsUp((Node *) newquery, ctx->levelsup, 1);
+ }
+
+ /*
+ * Here's where we do the actual substitution.
+ */
+ rte->rtekind = RTE_SUBQUERY;
+ rte->subquery = newquery;
+ rte->security_barrier = false;
+
+ ctx->refcount--;
+ }
+ }
+
+ query_tree_walker(query, inline_cte_walker, ctxp,
+ QTW_IGNORE_RANGE_TABLE | QTW_IGNORE_CTE_SUBQUERIES);
+
+ ctx->levelsup--;
+
+ return false;
+ }
+
+ return expression_tree_walker(node, inline_cte_walker, ctxp);
+}
+
+static void inline_cte(PlannerInfo *root, CommonTableExpr *cte)
+{
+ struct inline_cte_walker_ctx ctx;
+ ctx.ctequery = castNode(Query, cte->ctequery);
+ ctx.ctename = cte->ctename;
+ ctx.refcount = cte->cterefcount;
+ ctx.levelsup = -1;
+ ctx.cte = cte;
+ inline_cte_walker((Node *) root->parse, &ctx);
+ /* we must replace all references */
+ Assert(ctx.refcount == 0);
+}
+
+
/*
* SS_process_ctes: process a query's WITH list
*
@@ -1183,6 +1318,23 @@ SS_process_ctes(PlannerInfo *root)
continue;
}
+ /*
+ * Consider inlining the CTE rather than planning it separately.
+ *
+ * XXX this likely needs some additional checks.
+ */
+ if (cmdType == CMD_SELECT &&
+ !cte->ctematerialized &&
+ !cte->cterecursive &&
+ (castNode(Query, cte->ctequery)->rowMarks == NIL))
+ {
+ inline_cte(root, cte);
+
+ /* Make a dummy entry in cte_plan_ids */
+ root->cte_plan_ids = lappend_int(root->cte_plan_ids, -1);
+ continue;
+ }
+
/*
* Copy the source Query node. Probably not necessary, but let's keep
* this similar to make_subplan.
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 7855cff30d..047d3de71c 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -1397,6 +1397,7 @@ typedef struct CommonTableExpr
int location; /* token location, or -1 if unknown */
/* These fields are set during parse analysis: */
bool cterecursive; /* is this CTE actually recursive? */
+ bool ctematerialized; /* is this an optimization fence? */
int cterefcount; /* number of RTEs referencing this CTE
* (excluding internal self-references) */
List *ctecolnames; /* list of output column names */
diff --git a/src/test/regress/expected/rowsecurity.out b/src/test/regress/expected/rowsecurity.out
index bc16ca4c43..0d8d382830 100644
--- a/src/test/regress/expected/rowsecurity.out
+++ b/src/test/regress/expected/rowsecurity.out
@@ -2181,29 +2181,25 @@ EXPLAIN (COSTS OFF) EXECUTE plancache_test;
PREPARE plancache_test2 AS WITH q AS (SELECT * FROM z1 WHERE f_leak(b)) SELECT * FROM q,z2;
EXPLAIN (COSTS OFF) EXECUTE plancache_test2;
- QUERY PLAN
--------------------------------------------------
+ QUERY PLAN
+-----------------------------------------------------
Nested Loop
- CTE q
- -> Seq Scan on z1
- Filter: (((a % 2) = 0) AND f_leak(b))
- -> CTE Scan on q
+ -> Seq Scan on z2
-> Materialize
- -> Seq Scan on z2
-(7 rows)
+ -> Seq Scan on z1
+ Filter: (((a % 2) = 0) AND f_leak(b))
+(5 rows)
PREPARE plancache_test3 AS WITH q AS (SELECT * FROM z2) SELECT * FROM q,z1 WHERE f_leak(z1.b);
EXPLAIN (COSTS OFF) EXECUTE plancache_test3;
QUERY PLAN
-----------------------------------------------------
Nested Loop
- CTE q
- -> Seq Scan on z2
- -> CTE Scan on q
+ -> Seq Scan on z2
-> Materialize
-> Seq Scan on z1
Filter: (((a % 2) = 0) AND f_leak(b))
-(7 rows)
+(5 rows)
SET ROLE regress_rls_group1;
SELECT * FROM z1 WHERE f_leak(b);
@@ -2230,28 +2226,24 @@ EXPLAIN (COSTS OFF) EXECUTE plancache_test;
(2 rows)
EXPLAIN (COSTS OFF) EXECUTE plancache_test2;
- QUERY PLAN
--------------------------------------------------
+ QUERY PLAN
+-----------------------------------------------------
Nested Loop
- CTE q
- -> Seq Scan on z1
- Filter: (((a % 2) = 0) AND f_leak(b))
- -> CTE Scan on q
+ -> Seq Scan on z2
-> Materialize
- -> Seq Scan on z2
-(7 rows)
+ -> Seq Scan on z1
+ Filter: (((a % 2) = 0) AND f_leak(b))
+(5 rows)
EXPLAIN (COSTS OFF) EXECUTE plancache_test3;
QUERY PLAN
-----------------------------------------------------
Nested Loop
- CTE q
- -> Seq Scan on z2
- -> CTE Scan on q
+ -> Seq Scan on z2
-> Materialize
-> Seq Scan on z1
Filter: (((a % 2) = 0) AND f_leak(b))
-(7 rows)
+(5 rows)
SET SESSION AUTHORIZATION regress_rls_carol;
SELECT * FROM z1 WHERE f_leak(b);
@@ -2278,28 +2270,24 @@ EXPLAIN (COSTS OFF) EXECUTE plancache_test;
(2 rows)
EXPLAIN (COSTS OFF) EXECUTE plancache_test2;
- QUERY PLAN
--------------------------------------------------
+ QUERY PLAN
+-----------------------------------------------------
Nested Loop
- CTE q
- -> Seq Scan on z1
- Filter: (((a % 2) = 1) AND f_leak(b))
- -> CTE Scan on q
+ -> Seq Scan on z2
-> Materialize
- -> Seq Scan on z2
-(7 rows)
+ -> Seq Scan on z1
+ Filter: (((a % 2) = 1) AND f_leak(b))
+(5 rows)
EXPLAIN (COSTS OFF) EXECUTE plancache_test3;
QUERY PLAN
-----------------------------------------------------
Nested Loop
- CTE q
- -> Seq Scan on z2
- -> CTE Scan on q
+ -> Seq Scan on z2
-> Materialize
-> Seq Scan on z1
Filter: (((a % 2) = 1) AND f_leak(b))
-(7 rows)
+(5 rows)
SET ROLE regress_rls_group2;
SELECT * FROM z1 WHERE f_leak(b);
@@ -2326,28 +2314,24 @@ EXPLAIN (COSTS OFF) EXECUTE plancache_test;
(2 rows)
EXPLAIN (COSTS OFF) EXECUTE plancache_test2;
- QUERY PLAN
--------------------------------------------------
+ QUERY PLAN
+-----------------------------------------------------
Nested Loop
- CTE q
- -> Seq Scan on z1
- Filter: (((a % 2) = 1) AND f_leak(b))
- -> CTE Scan on q
+ -> Seq Scan on z2
-> Materialize
- -> Seq Scan on z2
-(7 rows)
+ -> Seq Scan on z1
+ Filter: (((a % 2) = 1) AND f_leak(b))
+(5 rows)
EXPLAIN (COSTS OFF) EXECUTE plancache_test3;
QUERY PLAN
-----------------------------------------------------
Nested Loop
- CTE q
- -> Seq Scan on z2
- -> CTE Scan on q
+ -> Seq Scan on z2
-> Materialize
-> Seq Scan on z1
Filter: (((a % 2) = 1) AND f_leak(b))
-(7 rows)
+(5 rows)
--
-- Views should follow policy for view owner.
@@ -2854,13 +2838,11 @@ NOTICE: f_leak => 98f13708210194c475687be6106a3b84
(11 rows)
EXPLAIN (COSTS OFF) WITH cte1 AS (SELECT * FROM t1 WHERE f_leak(b)) SELECT * FROM cte1;
- QUERY PLAN
--------------------------------------------------
- CTE Scan on cte1
- CTE cte1
- -> Seq Scan on t1
- Filter: (((a % 2) = 0) AND f_leak(b))
-(4 rows)
+ QUERY PLAN
+-----------------------------------------
+ Seq Scan on t1
+ Filter: (((a % 2) = 0) AND f_leak(b))
+(2 rows)
WITH cte1 AS (UPDATE t1 SET a = a + 1 RETURNING *) SELECT * FROM cte1; --fail
ERROR: new row violates row-level security policy for table "t1"
diff --git a/src/test/regress/expected/rowtypes.out b/src/test/regress/expected/rowtypes.out
index 30053d07df..c0df2f6333 100644
--- a/src/test/regress/expected/rowtypes.out
+++ b/src/test/regress/expected/rowtypes.out
@@ -1041,14 +1041,11 @@ with r(a,b) as
(values (1,row(1,2)), (1,row(null,null)), (1,null),
(null,row(1,2)), (null,row(null,null)), (null,null) )
select r, r is null as isnull, r is not null as isnotnull from r;
- QUERY PLAN
-----------------------------------------------------------
- CTE Scan on r
- Output: r.*, (r.* IS NULL), (r.* IS NOT NULL)
- CTE r
- -> Values Scan on "*VALUES*"
- Output: "*VALUES*".column1, "*VALUES*".column2
-(5 rows)
+ QUERY PLAN
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ Values Scan on "*VALUES*"
+ Output: ROW("*VALUES*".column1, "*VALUES*".column2), (("*VALUES*".column1 IS NULL) AND ("*VALUES*".column2 IS NOT DISTINCT FROM NULL)), (("*VALUES*".column1 IS NOT NULL) AND ("*VALUES*".column2 IS DISTINCT FROM NULL))
+(2 rows)
with r(a,b) as
(values (1,row(1,2)), (1,row(null,null)), (1,null),
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 744d501e31..882469b79f 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -3135,10 +3135,8 @@ RETURNING *;
Conflict Resolution: UPDATE
Conflict Arbiter Indexes: hat_data_unique_idx
Conflict Filter: ((excluded.hat_color <> 'forbidden'::bpchar) AND (hat_data.* <> excluded.*))
- CTE data
- -> Values Scan on "*VALUES*"
- -> CTE Scan on data
-(7 rows)
+ -> Values Scan on "*VALUES*"
+(5 rows)
SELECT * FROM hat_data WHERE hat_name IN ('h8', 'h9', 'h7') ORDER BY hat_name;
hat_name | hat_color
--
2.17.1
[text/plain] regression.diffs (970B, ../../20180725054236.GB17411@fetter.org/3-regression.diffs)
download | inline:
*** /home/shackle/pggit/postgresql/src/test/regress/expected/window.out 2018-07-24 22:14:09.015838790 -0700
--- /home/shackle/pggit/postgresql/src/test/regress/results/window.out 2018-07-24 22:20:36.766079578 -0700
***************
*** 3297,3303 ****
);
eq1 | eq2 | eq3
-----+-----+-----
! t | t | t
(1 row)
--
--- 3297,3303 ----
);
eq1 | eq2 | eq3
-----+-----+-----
! f | f | t
(1 row)
--
======================================================================
*** /home/shackle/pggit/postgresql/src/test/regress/expected/with.out 2018-07-24 22:14:09.017838781 -0700
--- /home/shackle/pggit/postgresql/src/test/regress/results/with.out 2018-07-24 22:20:37.562075966 -0700
***************
*** 18,24 ****
) ss;
count
-------
! 5
(1 row)
-- WITH RECURSIVE
--- 18,24 ----
) ss;
count
-------
! 10
(1 row)
-- WITH RECURSIVE
======================================================================
[text/plain] with.out (47.9K, ../../20180725054236.GB17411@fetter.org/4-with.out)
download | inline:
--
-- Tests for common table expressions (WITH query, ... SELECT ...)
--
-- Basic WITH
WITH q1(x,y) AS (SELECT 1,2)
SELECT * FROM q1, q1 AS q2;
x | y | x | y
---+---+---+---
1 | 2 | 1 | 2
(1 row)
-- Multiple uses are evaluated only once
SELECT count(*) FROM (
WITH q1(x) AS (SELECT random() FROM generate_series(1, 5))
SELECT * FROM q1
UNION
SELECT * FROM q1
) ss;
count
-------
5
(1 row)
-- WITH RECURSIVE
-- sum of 1..100
WITH RECURSIVE t(n) AS (
VALUES (1)
UNION ALL
SELECT n+1 FROM t WHERE n < 100
)
SELECT sum(n) FROM t;
sum
------
5050
(1 row)
WITH RECURSIVE t(n) AS (
SELECT (VALUES(1))
UNION ALL
SELECT n+1 FROM t WHERE n < 5
)
SELECT * FROM t;
n
---
1
2
3
4
5
(5 rows)
-- recursive view
CREATE RECURSIVE VIEW nums (n) AS
VALUES (1)
UNION ALL
SELECT n+1 FROM nums WHERE n < 5;
SELECT * FROM nums;
n
---
1
2
3
4
5
(5 rows)
CREATE OR REPLACE RECURSIVE VIEW nums (n) AS
VALUES (1)
UNION ALL
SELECT n+1 FROM nums WHERE n < 6;
SELECT * FROM nums;
n
---
1
2
3
4
5
6
(6 rows)
-- This is an infinite loop with UNION ALL, but not with UNION
WITH RECURSIVE t(n) AS (
SELECT 1
UNION
SELECT 10-n FROM t)
SELECT * FROM t;
n
---
1
9
(2 rows)
-- This'd be an infinite loop, but outside query reads only as much as needed
WITH RECURSIVE t(n) AS (
VALUES (1)
UNION ALL
SELECT n+1 FROM t)
SELECT * FROM t LIMIT 10;
n
----
1
2
3
4
5
6
7
8
9
10
(10 rows)
-- UNION case should have same property
WITH RECURSIVE t(n) AS (
SELECT 1
UNION
SELECT n+1 FROM t)
SELECT * FROM t LIMIT 10;
n
----
1
2
3
4
5
6
7
8
9
10
(10 rows)
-- Test behavior with an unknown-type literal in the WITH
WITH q AS (SELECT 'foo' AS x)
SELECT x, x IS OF (text) AS is_text FROM q;
x | is_text
-----+---------
foo | t
(1 row)
WITH RECURSIVE t(n) AS (
SELECT 'foo'
UNION ALL
SELECT n || ' bar' FROM t WHERE length(n) < 20
)
SELECT n, n IS OF (text) AS is_text FROM t;
n | is_text
-------------------------+---------
foo | t
foo bar | t
foo bar bar | t
foo bar bar bar | t
foo bar bar bar bar | t
foo bar bar bar bar bar | t
(6 rows)
-- In a perfect world, this would work and resolve the literal as int ...
-- but for now, we have to be content with resolving to text too soon.
WITH RECURSIVE t(n) AS (
SELECT '7'
UNION ALL
SELECT n+1 FROM t WHERE n < 10
)
SELECT n, n IS OF (int) AS is_int FROM t;
ERROR: operator does not exist: text + integer
LINE 4: SELECT n+1 FROM t WHERE n < 10
^
HINT: No operator matches the given name and argument types. You might need to add explicit type casts.
--
-- Some examples with a tree
--
-- department structure represented here is as follows:
--
-- ROOT-+->A-+->B-+->C
-- | |
-- | +->D-+->F
-- +->E-+->G
CREATE TEMP TABLE department (
id INTEGER PRIMARY KEY, -- department ID
parent_department INTEGER REFERENCES department, -- upper department ID
name TEXT -- department name
);
INSERT INTO department VALUES (0, NULL, 'ROOT');
INSERT INTO department VALUES (1, 0, 'A');
INSERT INTO department VALUES (2, 1, 'B');
INSERT INTO department VALUES (3, 2, 'C');
INSERT INTO department VALUES (4, 2, 'D');
INSERT INTO department VALUES (5, 0, 'E');
INSERT INTO department VALUES (6, 4, 'F');
INSERT INTO department VALUES (7, 5, 'G');
-- extract all departments under 'A'. Result should be A, B, C, D and F
WITH RECURSIVE subdepartment AS
(
-- non recursive term
SELECT name as root_name, * FROM department WHERE name = 'A'
UNION ALL
-- recursive term
SELECT sd.root_name, d.* FROM department AS d, subdepartment AS sd
WHERE d.parent_department = sd.id
)
SELECT * FROM subdepartment ORDER BY name;
root_name | id | parent_department | name
-----------+----+-------------------+------
A | 1 | 0 | A
A | 2 | 1 | B
A | 3 | 2 | C
A | 4 | 2 | D
A | 6 | 4 | F
(5 rows)
-- extract all departments under 'A' with "level" number
WITH RECURSIVE subdepartment(level, id, parent_department, name) AS
(
-- non recursive term
SELECT 1, * FROM department WHERE name = 'A'
UNION ALL
-- recursive term
SELECT sd.level + 1, d.* FROM department AS d, subdepartment AS sd
WHERE d.parent_department = sd.id
)
SELECT * FROM subdepartment ORDER BY name;
level | id | parent_department | name
-------+----+-------------------+------
1 | 1 | 0 | A
2 | 2 | 1 | B
3 | 3 | 2 | C
3 | 4 | 2 | D
4 | 6 | 4 | F
(5 rows)
-- extract all departments under 'A' with "level" number.
-- Only shows level 2 or more
WITH RECURSIVE subdepartment(level, id, parent_department, name) AS
(
-- non recursive term
SELECT 1, * FROM department WHERE name = 'A'
UNION ALL
-- recursive term
SELECT sd.level + 1, d.* FROM department AS d, subdepartment AS sd
WHERE d.parent_department = sd.id
)
SELECT * FROM subdepartment WHERE level >= 2 ORDER BY name;
level | id | parent_department | name
-------+----+-------------------+------
2 | 2 | 1 | B
3 | 3 | 2 | C
3 | 4 | 2 | D
4 | 6 | 4 | F
(4 rows)
-- "RECURSIVE" is ignored if the query has no self-reference
WITH RECURSIVE subdepartment AS
(
-- note lack of recursive UNION structure
SELECT * FROM department WHERE name = 'A'
)
SELECT * FROM subdepartment ORDER BY name;
id | parent_department | name
----+-------------------+------
1 | 0 | A
(1 row)
-- inside subqueries
SELECT count(*) FROM (
WITH RECURSIVE t(n) AS (
SELECT 1 UNION ALL SELECT n + 1 FROM t WHERE n < 500
)
SELECT * FROM t) AS t WHERE n < (
SELECT count(*) FROM (
WITH RECURSIVE t(n) AS (
SELECT 1 UNION ALL SELECT n + 1 FROM t WHERE n < 100
)
SELECT * FROM t WHERE n < 50000
) AS t WHERE n < 100);
count
-------
98
(1 row)
-- use same CTE twice at different subquery levels
WITH q1(x,y) AS (
SELECT hundred, sum(ten) FROM tenk1 GROUP BY hundred
)
SELECT count(*) FROM q1 WHERE y > (SELECT sum(y)/100 FROM q1 qsub);
count
-------
50
(1 row)
-- via a VIEW
CREATE TEMPORARY VIEW vsubdepartment AS
WITH RECURSIVE subdepartment AS
(
-- non recursive term
SELECT * FROM department WHERE name = 'A'
UNION ALL
-- recursive term
SELECT d.* FROM department AS d, subdepartment AS sd
WHERE d.parent_department = sd.id
)
SELECT * FROM subdepartment;
SELECT * FROM vsubdepartment ORDER BY name;
id | parent_department | name
----+-------------------+------
1 | 0 | A
2 | 1 | B
3 | 2 | C
4 | 2 | D
6 | 4 | F
(5 rows)
-- Check reverse listing
SELECT pg_get_viewdef('vsubdepartment'::regclass);
pg_get_viewdef
-----------------------------------------------
WITH RECURSIVE subdepartment AS ( +
SELECT department.id, +
department.parent_department, +
department.name +
FROM department +
WHERE (department.name = 'A'::text)+
UNION ALL +
SELECT d.id, +
d.parent_department, +
d.name +
FROM department d, +
subdepartment sd +
WHERE (d.parent_department = sd.id)+
) +
SELECT subdepartment.id, +
subdepartment.parent_department, +
subdepartment.name +
FROM subdepartment;
(1 row)
SELECT pg_get_viewdef('vsubdepartment'::regclass, true);
pg_get_viewdef
---------------------------------------------
WITH RECURSIVE subdepartment AS ( +
SELECT department.id, +
department.parent_department, +
department.name +
FROM department +
WHERE department.name = 'A'::text+
UNION ALL +
SELECT d.id, +
d.parent_department, +
d.name +
FROM department d, +
subdepartment sd +
WHERE d.parent_department = sd.id+
) +
SELECT subdepartment.id, +
subdepartment.parent_department, +
subdepartment.name +
FROM subdepartment;
(1 row)
-- Another reverse-listing example
CREATE VIEW sums_1_100 AS
WITH RECURSIVE t(n) AS (
VALUES (1)
UNION ALL
SELECT n+1 FROM t WHERE n < 100
)
SELECT sum(n) FROM t;
\d+ sums_1_100
View "public.sums_1_100"
Column | Type | Collation | Nullable | Default | Storage | Description
--------+--------+-----------+----------+---------+---------+-------------
sum | bigint | | | | plain |
View definition:
WITH RECURSIVE t(n) AS (
VALUES (1)
UNION ALL
SELECT t_1.n + 1
FROM t t_1
WHERE t_1.n < 100
)
SELECT sum(t.n) AS sum
FROM t;
-- corner case in which sub-WITH gets initialized first
with recursive q as (
select * from department
union all
(with x as (select * from q)
select * from x)
)
select * from q limit 24;
id | parent_department | name
----+-------------------+------
0 | | ROOT
1 | 0 | A
2 | 1 | B
3 | 2 | C
4 | 2 | D
5 | 0 | E
6 | 4 | F
7 | 5 | G
0 | | ROOT
1 | 0 | A
2 | 1 | B
3 | 2 | C
4 | 2 | D
5 | 0 | E
6 | 4 | F
7 | 5 | G
0 | | ROOT
1 | 0 | A
2 | 1 | B
3 | 2 | C
4 | 2 | D
5 | 0 | E
6 | 4 | F
7 | 5 | G
(24 rows)
with recursive q as (
select * from department
union all
(with recursive x as (
select * from department
union all
(select * from q union all select * from x)
)
select * from x)
)
select * from q limit 32;
id | parent_department | name
----+-------------------+------
0 | | ROOT
1 | 0 | A
2 | 1 | B
3 | 2 | C
4 | 2 | D
5 | 0 | E
6 | 4 | F
7 | 5 | G
0 | | ROOT
1 | 0 | A
2 | 1 | B
3 | 2 | C
4 | 2 | D
5 | 0 | E
6 | 4 | F
7 | 5 | G
0 | | ROOT
1 | 0 | A
2 | 1 | B
3 | 2 | C
4 | 2 | D
5 | 0 | E
6 | 4 | F
7 | 5 | G
0 | | ROOT
1 | 0 | A
2 | 1 | B
3 | 2 | C
4 | 2 | D
5 | 0 | E
6 | 4 | F
7 | 5 | G
(32 rows)
-- recursive term has sub-UNION
WITH RECURSIVE t(i,j) AS (
VALUES (1,2)
UNION ALL
SELECT t2.i, t.j+1 FROM
(SELECT 2 AS i UNION ALL SELECT 3 AS i) AS t2
JOIN t ON (t2.i = t.i+1))
SELECT * FROM t;
i | j
---+---
1 | 2
2 | 3
3 | 4
(3 rows)
--
-- different tree example
--
CREATE TEMPORARY TABLE tree(
id INTEGER PRIMARY KEY,
parent_id INTEGER REFERENCES tree(id)
);
INSERT INTO tree
VALUES (1, NULL), (2, 1), (3,1), (4,2), (5,2), (6,2), (7,3), (8,3),
(9,4), (10,4), (11,7), (12,7), (13,7), (14, 9), (15,11), (16,11);
--
-- get all paths from "second level" nodes to leaf nodes
--
WITH RECURSIVE t(id, path) AS (
VALUES(1,ARRAY[]::integer[])
UNION ALL
SELECT tree.id, t.path || tree.id
FROM tree JOIN t ON (tree.parent_id = t.id)
)
SELECT t1.*, t2.* FROM t AS t1 JOIN t AS t2 ON
(t1.path[1] = t2.path[1] AND
array_upper(t1.path,1) = 1 AND
array_upper(t2.path,1) > 1)
ORDER BY t1.id, t2.id;
id | path | id | path
----+------+----+-------------
2 | {2} | 4 | {2,4}
2 | {2} | 5 | {2,5}
2 | {2} | 6 | {2,6}
2 | {2} | 9 | {2,4,9}
2 | {2} | 10 | {2,4,10}
2 | {2} | 14 | {2,4,9,14}
3 | {3} | 7 | {3,7}
3 | {3} | 8 | {3,8}
3 | {3} | 11 | {3,7,11}
3 | {3} | 12 | {3,7,12}
3 | {3} | 13 | {3,7,13}
3 | {3} | 15 | {3,7,11,15}
3 | {3} | 16 | {3,7,11,16}
(13 rows)
-- just count 'em
WITH RECURSIVE t(id, path) AS (
VALUES(1,ARRAY[]::integer[])
UNION ALL
SELECT tree.id, t.path || tree.id
FROM tree JOIN t ON (tree.parent_id = t.id)
)
SELECT t1.id, count(t2.*) FROM t AS t1 JOIN t AS t2 ON
(t1.path[1] = t2.path[1] AND
array_upper(t1.path,1) = 1 AND
array_upper(t2.path,1) > 1)
GROUP BY t1.id
ORDER BY t1.id;
id | count
----+-------
2 | 6
3 | 7
(2 rows)
-- this variant tickled a whole-row-variable bug in 8.4devel
WITH RECURSIVE t(id, path) AS (
VALUES(1,ARRAY[]::integer[])
UNION ALL
SELECT tree.id, t.path || tree.id
FROM tree JOIN t ON (tree.parent_id = t.id)
)
SELECT t1.id, t2.path, t2 FROM t AS t1 JOIN t AS t2 ON
(t1.id=t2.id);
id | path | t2
----+-------------+--------------------
1 | {} | (1,{})
2 | {2} | (2,{2})
3 | {3} | (3,{3})
4 | {2,4} | (4,"{2,4}")
5 | {2,5} | (5,"{2,5}")
6 | {2,6} | (6,"{2,6}")
7 | {3,7} | (7,"{3,7}")
8 | {3,8} | (8,"{3,8}")
9 | {2,4,9} | (9,"{2,4,9}")
10 | {2,4,10} | (10,"{2,4,10}")
11 | {3,7,11} | (11,"{3,7,11}")
12 | {3,7,12} | (12,"{3,7,12}")
13 | {3,7,13} | (13,"{3,7,13}")
14 | {2,4,9,14} | (14,"{2,4,9,14}")
15 | {3,7,11,15} | (15,"{3,7,11,15}")
16 | {3,7,11,16} | (16,"{3,7,11,16}")
(16 rows)
--
-- test cycle detection
--
create temp table graph( f int, t int, label text );
insert into graph values
(1, 2, 'arc 1 -> 2'),
(1, 3, 'arc 1 -> 3'),
(2, 3, 'arc 2 -> 3'),
(1, 4, 'arc 1 -> 4'),
(4, 5, 'arc 4 -> 5'),
(5, 1, 'arc 5 -> 1');
with recursive search_graph(f, t, label, path, cycle) as (
select *, array[row(g.f, g.t)], false from graph g
union all
select g.*, path || row(g.f, g.t), row(g.f, g.t) = any(path)
from graph g, search_graph sg
where g.f = sg.t and not cycle
)
select * from search_graph;
f | t | label | path | cycle
---+---+------------+-------------------------------------------+-------
1 | 2 | arc 1 -> 2 | {"(1,2)"} | f
1 | 3 | arc 1 -> 3 | {"(1,3)"} | f
2 | 3 | arc 2 -> 3 | {"(2,3)"} | f
1 | 4 | arc 1 -> 4 | {"(1,4)"} | f
4 | 5 | arc 4 -> 5 | {"(4,5)"} | f
5 | 1 | arc 5 -> 1 | {"(5,1)"} | f
1 | 2 | arc 1 -> 2 | {"(5,1)","(1,2)"} | f
1 | 3 | arc 1 -> 3 | {"(5,1)","(1,3)"} | f
1 | 4 | arc 1 -> 4 | {"(5,1)","(1,4)"} | f
2 | 3 | arc 2 -> 3 | {"(1,2)","(2,3)"} | f
4 | 5 | arc 4 -> 5 | {"(1,4)","(4,5)"} | f
5 | 1 | arc 5 -> 1 | {"(4,5)","(5,1)"} | f
1 | 2 | arc 1 -> 2 | {"(4,5)","(5,1)","(1,2)"} | f
1 | 3 | arc 1 -> 3 | {"(4,5)","(5,1)","(1,3)"} | f
1 | 4 | arc 1 -> 4 | {"(4,5)","(5,1)","(1,4)"} | f
2 | 3 | arc 2 -> 3 | {"(5,1)","(1,2)","(2,3)"} | f
4 | 5 | arc 4 -> 5 | {"(5,1)","(1,4)","(4,5)"} | f
5 | 1 | arc 5 -> 1 | {"(1,4)","(4,5)","(5,1)"} | f
1 | 2 | arc 1 -> 2 | {"(1,4)","(4,5)","(5,1)","(1,2)"} | f
1 | 3 | arc 1 -> 3 | {"(1,4)","(4,5)","(5,1)","(1,3)"} | f
1 | 4 | arc 1 -> 4 | {"(1,4)","(4,5)","(5,1)","(1,4)"} | t
2 | 3 | arc 2 -> 3 | {"(4,5)","(5,1)","(1,2)","(2,3)"} | f
4 | 5 | arc 4 -> 5 | {"(4,5)","(5,1)","(1,4)","(4,5)"} | t
5 | 1 | arc 5 -> 1 | {"(5,1)","(1,4)","(4,5)","(5,1)"} | t
2 | 3 | arc 2 -> 3 | {"(1,4)","(4,5)","(5,1)","(1,2)","(2,3)"} | f
(25 rows)
-- ordering by the path column has same effect as SEARCH DEPTH FIRST
with recursive search_graph(f, t, label, path, cycle) as (
select *, array[row(g.f, g.t)], false from graph g
union all
select g.*, path || row(g.f, g.t), row(g.f, g.t) = any(path)
from graph g, search_graph sg
where g.f = sg.t and not cycle
)
select * from search_graph order by path;
f | t | label | path | cycle
---+---+------------+-------------------------------------------+-------
1 | 2 | arc 1 -> 2 | {"(1,2)"} | f
2 | 3 | arc 2 -> 3 | {"(1,2)","(2,3)"} | f
1 | 3 | arc 1 -> 3 | {"(1,3)"} | f
1 | 4 | arc 1 -> 4 | {"(1,4)"} | f
4 | 5 | arc 4 -> 5 | {"(1,4)","(4,5)"} | f
5 | 1 | arc 5 -> 1 | {"(1,4)","(4,5)","(5,1)"} | f
1 | 2 | arc 1 -> 2 | {"(1,4)","(4,5)","(5,1)","(1,2)"} | f
2 | 3 | arc 2 -> 3 | {"(1,4)","(4,5)","(5,1)","(1,2)","(2,3)"} | f
1 | 3 | arc 1 -> 3 | {"(1,4)","(4,5)","(5,1)","(1,3)"} | f
1 | 4 | arc 1 -> 4 | {"(1,4)","(4,5)","(5,1)","(1,4)"} | t
2 | 3 | arc 2 -> 3 | {"(2,3)"} | f
4 | 5 | arc 4 -> 5 | {"(4,5)"} | f
5 | 1 | arc 5 -> 1 | {"(4,5)","(5,1)"} | f
1 | 2 | arc 1 -> 2 | {"(4,5)","(5,1)","(1,2)"} | f
2 | 3 | arc 2 -> 3 | {"(4,5)","(5,1)","(1,2)","(2,3)"} | f
1 | 3 | arc 1 -> 3 | {"(4,5)","(5,1)","(1,3)"} | f
1 | 4 | arc 1 -> 4 | {"(4,5)","(5,1)","(1,4)"} | f
4 | 5 | arc 4 -> 5 | {"(4,5)","(5,1)","(1,4)","(4,5)"} | t
5 | 1 | arc 5 -> 1 | {"(5,1)"} | f
1 | 2 | arc 1 -> 2 | {"(5,1)","(1,2)"} | f
2 | 3 | arc 2 -> 3 | {"(5,1)","(1,2)","(2,3)"} | f
1 | 3 | arc 1 -> 3 | {"(5,1)","(1,3)"} | f
1 | 4 | arc 1 -> 4 | {"(5,1)","(1,4)"} | f
4 | 5 | arc 4 -> 5 | {"(5,1)","(1,4)","(4,5)"} | f
5 | 1 | arc 5 -> 1 | {"(5,1)","(1,4)","(4,5)","(5,1)"} | t
(25 rows)
--
-- test multiple WITH queries
--
WITH RECURSIVE
y (id) AS (VALUES (1)),
x (id) AS (SELECT * FROM y UNION ALL SELECT id+1 FROM x WHERE id < 5)
SELECT * FROM x;
id
----
1
2
3
4
5
(5 rows)
-- forward reference OK
WITH RECURSIVE
x(id) AS (SELECT * FROM y UNION ALL SELECT id+1 FROM x WHERE id < 5),
y(id) AS (values (1))
SELECT * FROM x;
id
----
1
2
3
4
5
(5 rows)
WITH RECURSIVE
x(id) AS
(VALUES (1) UNION ALL SELECT id+1 FROM x WHERE id < 5),
y(id) AS
(VALUES (1) UNION ALL SELECT id+1 FROM y WHERE id < 10)
SELECT y.*, x.* FROM y LEFT JOIN x USING (id);
id | id
----+----
1 | 1
2 | 2
3 | 3
4 | 4
5 | 5
6 |
7 |
8 |
9 |
10 |
(10 rows)
WITH RECURSIVE
x(id) AS
(VALUES (1) UNION ALL SELECT id+1 FROM x WHERE id < 5),
y(id) AS
(VALUES (1) UNION ALL SELECT id+1 FROM x WHERE id < 10)
SELECT y.*, x.* FROM y LEFT JOIN x USING (id);
id | id
----+----
1 | 1
2 | 2
3 | 3
4 | 4
5 | 5
6 |
(6 rows)
WITH RECURSIVE
x(id) AS
(SELECT 1 UNION ALL SELECT id+1 FROM x WHERE id < 3 ),
y(id) AS
(SELECT * FROM x UNION ALL SELECT * FROM x),
z(id) AS
(SELECT * FROM x UNION ALL SELECT id+1 FROM z WHERE id < 10)
SELECT * FROM z;
id
----
1
2
3
2
3
4
3
4
5
4
5
6
5
6
7
6
7
8
7
8
9
8
9
10
9
10
10
(27 rows)
WITH RECURSIVE
x(id) AS
(SELECT 1 UNION ALL SELECT id+1 FROM x WHERE id < 3 ),
y(id) AS
(SELECT * FROM x UNION ALL SELECT * FROM x),
z(id) AS
(SELECT * FROM y UNION ALL SELECT id+1 FROM z WHERE id < 10)
SELECT * FROM z;
id
----
1
2
3
1
2
3
2
3
4
2
3
4
3
4
5
3
4
5
4
5
6
4
5
6
5
6
7
5
6
7
6
7
8
6
7
8
7
8
9
7
8
9
8
9
10
8
9
10
9
10
9
10
10
10
(54 rows)
--
-- Test WITH attached to a data-modifying statement
--
CREATE TEMPORARY TABLE y (a INTEGER);
INSERT INTO y SELECT generate_series(1, 10);
WITH t AS (
SELECT a FROM y
)
INSERT INTO y
SELECT a+20 FROM t RETURNING *;
a
----
21
22
23
24
25
26
27
28
29
30
(10 rows)
SELECT * FROM y;
a
----
1
2
3
4
5
6
7
8
9
10
21
22
23
24
25
26
27
28
29
30
(20 rows)
WITH t AS (
SELECT a FROM y
)
UPDATE y SET a = y.a-10 FROM t WHERE y.a > 20 AND t.a = y.a RETURNING y.a;
a
----
11
12
13
14
15
16
17
18
19
20
(10 rows)
SELECT * FROM y;
a
----
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
(20 rows)
WITH RECURSIVE t(a) AS (
SELECT 11
UNION ALL
SELECT a+1 FROM t WHERE a < 50
)
DELETE FROM y USING t WHERE t.a = y.a RETURNING y.a;
a
----
11
12
13
14
15
16
17
18
19
20
(10 rows)
SELECT * FROM y;
a
----
1
2
3
4
5
6
7
8
9
10
(10 rows)
DROP TABLE y;
--
-- error cases
--
-- INTERSECT
WITH RECURSIVE x(n) AS (SELECT 1 INTERSECT SELECT n+1 FROM x)
SELECT * FROM x;
ERROR: recursive query "x" does not have the form non-recursive-term UNION [ALL] recursive-term
LINE 1: WITH RECURSIVE x(n) AS (SELECT 1 INTERSECT SELECT n+1 FROM x...
^
WITH RECURSIVE x(n) AS (SELECT 1 INTERSECT ALL SELECT n+1 FROM x)
SELECT * FROM x;
ERROR: recursive query "x" does not have the form non-recursive-term UNION [ALL] recursive-term
LINE 1: WITH RECURSIVE x(n) AS (SELECT 1 INTERSECT ALL SELECT n+1 FR...
^
-- EXCEPT
WITH RECURSIVE x(n) AS (SELECT 1 EXCEPT SELECT n+1 FROM x)
SELECT * FROM x;
ERROR: recursive query "x" does not have the form non-recursive-term UNION [ALL] recursive-term
LINE 1: WITH RECURSIVE x(n) AS (SELECT 1 EXCEPT SELECT n+1 FROM x)
^
WITH RECURSIVE x(n) AS (SELECT 1 EXCEPT ALL SELECT n+1 FROM x)
SELECT * FROM x;
ERROR: recursive query "x" does not have the form non-recursive-term UNION [ALL] recursive-term
LINE 1: WITH RECURSIVE x(n) AS (SELECT 1 EXCEPT ALL SELECT n+1 FROM ...
^
-- no non-recursive term
WITH RECURSIVE x(n) AS (SELECT n FROM x)
SELECT * FROM x;
ERROR: recursive query "x" does not have the form non-recursive-term UNION [ALL] recursive-term
LINE 1: WITH RECURSIVE x(n) AS (SELECT n FROM x)
^
-- recursive term in the left hand side (strictly speaking, should allow this)
WITH RECURSIVE x(n) AS (SELECT n FROM x UNION ALL SELECT 1)
SELECT * FROM x;
ERROR: recursive reference to query "x" must not appear within its non-recursive term
LINE 1: WITH RECURSIVE x(n) AS (SELECT n FROM x UNION ALL SELECT 1)
^
CREATE TEMPORARY TABLE y (a INTEGER);
INSERT INTO y SELECT generate_series(1, 10);
-- LEFT JOIN
WITH RECURSIVE x(n) AS (SELECT a FROM y WHERE a = 1
UNION ALL
SELECT x.n+1 FROM y LEFT JOIN x ON x.n = y.a WHERE n < 10)
SELECT * FROM x;
ERROR: recursive reference to query "x" must not appear within an outer join
LINE 3: SELECT x.n+1 FROM y LEFT JOIN x ON x.n = y.a WHERE n < 10)
^
-- RIGHT JOIN
WITH RECURSIVE x(n) AS (SELECT a FROM y WHERE a = 1
UNION ALL
SELECT x.n+1 FROM x RIGHT JOIN y ON x.n = y.a WHERE n < 10)
SELECT * FROM x;
ERROR: recursive reference to query "x" must not appear within an outer join
LINE 3: SELECT x.n+1 FROM x RIGHT JOIN y ON x.n = y.a WHERE n < 10)
^
-- FULL JOIN
WITH RECURSIVE x(n) AS (SELECT a FROM y WHERE a = 1
UNION ALL
SELECT x.n+1 FROM x FULL JOIN y ON x.n = y.a WHERE n < 10)
SELECT * FROM x;
ERROR: recursive reference to query "x" must not appear within an outer join
LINE 3: SELECT x.n+1 FROM x FULL JOIN y ON x.n = y.a WHERE n < 10)
^
-- subquery
WITH RECURSIVE x(n) AS (SELECT 1 UNION ALL SELECT n+1 FROM x
WHERE n IN (SELECT * FROM x))
SELECT * FROM x;
ERROR: recursive reference to query "x" must not appear within a subquery
LINE 2: WHERE n IN (SELECT * FROM x))
^
-- aggregate functions
WITH RECURSIVE x(n) AS (SELECT 1 UNION ALL SELECT count(*) FROM x)
SELECT * FROM x;
ERROR: aggregate functions are not allowed in a recursive query's recursive term
LINE 1: WITH RECURSIVE x(n) AS (SELECT 1 UNION ALL SELECT count(*) F...
^
WITH RECURSIVE x(n) AS (SELECT 1 UNION ALL SELECT sum(n) FROM x)
SELECT * FROM x;
ERROR: aggregate functions are not allowed in a recursive query's recursive term
LINE 1: WITH RECURSIVE x(n) AS (SELECT 1 UNION ALL SELECT sum(n) FRO...
^
-- ORDER BY
WITH RECURSIVE x(n) AS (SELECT 1 UNION ALL SELECT n+1 FROM x ORDER BY 1)
SELECT * FROM x;
ERROR: ORDER BY in a recursive query is not implemented
LINE 1: ...VE x(n) AS (SELECT 1 UNION ALL SELECT n+1 FROM x ORDER BY 1)
^
-- LIMIT/OFFSET
WITH RECURSIVE x(n) AS (SELECT 1 UNION ALL SELECT n+1 FROM x LIMIT 10 OFFSET 1)
SELECT * FROM x;
ERROR: OFFSET in a recursive query is not implemented
LINE 1: ... AS (SELECT 1 UNION ALL SELECT n+1 FROM x LIMIT 10 OFFSET 1)
^
-- FOR UPDATE
WITH RECURSIVE x(n) AS (SELECT 1 UNION ALL SELECT n+1 FROM x FOR UPDATE)
SELECT * FROM x;
ERROR: FOR UPDATE/SHARE in a recursive query is not implemented
-- target list has a recursive query name
WITH RECURSIVE x(id) AS (values (1)
UNION ALL
SELECT (SELECT * FROM x) FROM x WHERE id < 5
) SELECT * FROM x;
ERROR: recursive reference to query "x" must not appear within a subquery
LINE 3: SELECT (SELECT * FROM x) FROM x WHERE id < 5
^
-- mutual recursive query (not implemented)
WITH RECURSIVE
x (id) AS (SELECT 1 UNION ALL SELECT id+1 FROM y WHERE id < 5),
y (id) AS (SELECT 1 UNION ALL SELECT id+1 FROM x WHERE id < 5)
SELECT * FROM x;
ERROR: mutual recursion between WITH items is not implemented
LINE 2: x (id) AS (SELECT 1 UNION ALL SELECT id+1 FROM y WHERE id ...
^
-- non-linear recursion is not allowed
WITH RECURSIVE foo(i) AS
(values (1)
UNION ALL
(SELECT i+1 FROM foo WHERE i < 10
UNION ALL
SELECT i+1 FROM foo WHERE i < 5)
) SELECT * FROM foo;
ERROR: recursive reference to query "foo" must not appear more than once
LINE 6: SELECT i+1 FROM foo WHERE i < 5)
^
WITH RECURSIVE foo(i) AS
(values (1)
UNION ALL
SELECT * FROM
(SELECT i+1 FROM foo WHERE i < 10
UNION ALL
SELECT i+1 FROM foo WHERE i < 5) AS t
) SELECT * FROM foo;
ERROR: recursive reference to query "foo" must not appear more than once
LINE 7: SELECT i+1 FROM foo WHERE i < 5) AS t
^
WITH RECURSIVE foo(i) AS
(values (1)
UNION ALL
(SELECT i+1 FROM foo WHERE i < 10
EXCEPT
SELECT i+1 FROM foo WHERE i < 5)
) SELECT * FROM foo;
ERROR: recursive reference to query "foo" must not appear within EXCEPT
LINE 6: SELECT i+1 FROM foo WHERE i < 5)
^
WITH RECURSIVE foo(i) AS
(values (1)
UNION ALL
(SELECT i+1 FROM foo WHERE i < 10
INTERSECT
SELECT i+1 FROM foo WHERE i < 5)
) SELECT * FROM foo;
ERROR: recursive reference to query "foo" must not appear more than once
LINE 6: SELECT i+1 FROM foo WHERE i < 5)
^
-- Wrong type induced from non-recursive term
WITH RECURSIVE foo(i) AS
(SELECT i FROM (VALUES(1),(2)) t(i)
UNION ALL
SELECT (i+1)::numeric(10,0) FROM foo WHERE i < 10)
SELECT * FROM foo;
ERROR: recursive query "foo" column 1 has type integer in non-recursive term but type numeric overall
LINE 2: (SELECT i FROM (VALUES(1),(2)) t(i)
^
HINT: Cast the output of the non-recursive term to the correct type.
-- rejects different typmod, too (should we allow this?)
WITH RECURSIVE foo(i) AS
(SELECT i::numeric(3,0) FROM (VALUES(1),(2)) t(i)
UNION ALL
SELECT (i+1)::numeric(10,0) FROM foo WHERE i < 10)
SELECT * FROM foo;
ERROR: recursive query "foo" column 1 has type numeric(3,0) in non-recursive term but type numeric overall
LINE 2: (SELECT i::numeric(3,0) FROM (VALUES(1),(2)) t(i)
^
HINT: Cast the output of the non-recursive term to the correct type.
-- disallow OLD/NEW reference in CTE
CREATE TEMPORARY TABLE x (n integer);
CREATE RULE r2 AS ON UPDATE TO x DO INSTEAD
WITH t AS (SELECT OLD.*) UPDATE y SET a = t.n FROM t;
ERROR: cannot refer to OLD within WITH query
--
-- test for bug #4902
--
with cte(foo) as ( values(42) ) values((select foo from cte));
column1
---------
42
(1 row)
with cte(foo) as ( select 42 ) select * from ((select foo from cte)) q;
foo
-----
42
(1 row)
-- test CTE referencing an outer-level variable (to see that changed-parameter
-- signaling still works properly after fixing this bug)
select ( with cte(foo) as ( values(f1) )
select (select foo from cte) )
from int4_tbl;
foo
-------------
0
123456
-123456
2147483647
-2147483647
(5 rows)
select ( with cte(foo) as ( values(f1) )
values((select foo from cte)) )
from int4_tbl;
column1
-------------
0
123456
-123456
2147483647
-2147483647
(5 rows)
--
-- test for nested-recursive-WITH bug
--
WITH RECURSIVE t(j) AS (
WITH RECURSIVE s(i) AS (
VALUES (1)
UNION ALL
SELECT i+1 FROM s WHERE i < 10
)
SELECT i FROM s
UNION ALL
SELECT j+1 FROM t WHERE j < 10
)
SELECT * FROM t;
j
----
1
2
3
4
5
6
7
8
9
10
2
3
4
5
6
7
8
9
10
3
4
5
6
7
8
9
10
4
5
6
7
8
9
10
5
6
7
8
9
10
6
7
8
9
10
7
8
9
10
8
9
10
9
10
10
(55 rows)
--
-- test WITH attached to intermediate-level set operation
--
WITH outermost(x) AS (
SELECT 1
UNION (WITH innermost as (SELECT 2)
SELECT * FROM innermost
UNION SELECT 3)
)
SELECT * FROM outermost ORDER BY 1;
x
---
1
2
3
(3 rows)
WITH outermost(x) AS (
SELECT 1
UNION (WITH innermost as (SELECT 2)
SELECT * FROM outermost -- fail
UNION SELECT * FROM innermost)
)
SELECT * FROM outermost ORDER BY 1;
ERROR: relation "outermost" does not exist
LINE 4: SELECT * FROM outermost
^
DETAIL: There is a WITH item named "outermost", but it cannot be referenced from this part of the query.
HINT: Use WITH RECURSIVE, or re-order the WITH items to remove forward references.
WITH RECURSIVE outermost(x) AS (
SELECT 1
UNION (WITH innermost as (SELECT 2)
SELECT * FROM outermost
UNION SELECT * FROM innermost)
)
SELECT * FROM outermost ORDER BY 1;
x
---
1
2
(2 rows)
WITH RECURSIVE outermost(x) AS (
WITH innermost as (SELECT 2 FROM outermost) -- fail
SELECT * FROM innermost
UNION SELECT * from outermost
)
SELECT * FROM outermost ORDER BY 1;
ERROR: recursive reference to query "outermost" must not appear within a subquery
LINE 2: WITH innermost as (SELECT 2 FROM outermost)
^
--
-- This test will fail with the old implementation of PARAM_EXEC parameter
-- assignment, because the "q1" Var passed down to A's targetlist subselect
-- looks exactly like the "A.id" Var passed down to C's subselect, causing
-- the old code to give them the same runtime PARAM_EXEC slot. But the
-- lifespans of the two parameters overlap, thanks to B also reading A.
--
with
A as ( select q2 as id, (select q1) as x from int8_tbl ),
B as ( select id, row_number() over (partition by id) as r from A ),
C as ( select A.id, array(select B.id from B where B.id = A.id) from A )
select * from C;
id | array
-------------------+-------------------------------------
456 | {456}
4567890123456789 | {4567890123456789,4567890123456789}
123 | {123}
4567890123456789 | {4567890123456789,4567890123456789}
-4567890123456789 | {-4567890123456789}
(5 rows)
--
-- Test CTEs read in non-initialization orders
--
WITH RECURSIVE
tab(id_key,link) AS (VALUES (1,17), (2,17), (3,17), (4,17), (6,17), (5,17)),
iter (id_key, row_type, link) AS (
SELECT 0, 'base', 17
UNION ALL (
WITH remaining(id_key, row_type, link, min) AS (
SELECT tab.id_key, 'true'::text, iter.link, MIN(tab.id_key) OVER ()
FROM tab INNER JOIN iter USING (link)
WHERE tab.id_key > iter.id_key
),
first_remaining AS (
SELECT id_key, row_type, link
FROM remaining
WHERE id_key=min
),
effect AS (
SELECT tab.id_key, 'new'::text, tab.link
FROM first_remaining e INNER JOIN tab ON e.id_key=tab.id_key
WHERE e.row_type = 'false'
)
SELECT * FROM first_remaining
UNION ALL SELECT * FROM effect
)
)
SELECT * FROM iter;
id_key | row_type | link
--------+----------+------
0 | base | 17
1 | true | 17
2 | true | 17
3 | true | 17
4 | true | 17
5 | true | 17
6 | true | 17
(7 rows)
WITH RECURSIVE
tab(id_key,link) AS (VALUES (1,17), (2,17), (3,17), (4,17), (6,17), (5,17)),
iter (id_key, row_type, link) AS (
SELECT 0, 'base', 17
UNION (
WITH remaining(id_key, row_type, link, min) AS (
SELECT tab.id_key, 'true'::text, iter.link, MIN(tab.id_key) OVER ()
FROM tab INNER JOIN iter USING (link)
WHERE tab.id_key > iter.id_key
),
first_remaining AS (
SELECT id_key, row_type, link
FROM remaining
WHERE id_key=min
),
effect AS (
SELECT tab.id_key, 'new'::text, tab.link
FROM first_remaining e INNER JOIN tab ON e.id_key=tab.id_key
WHERE e.row_type = 'false'
)
SELECT * FROM first_remaining
UNION ALL SELECT * FROM effect
)
)
SELECT * FROM iter;
id_key | row_type | link
--------+----------+------
0 | base | 17
1 | true | 17
2 | true | 17
3 | true | 17
4 | true | 17
5 | true | 17
6 | true | 17
(7 rows)
--
-- Data-modifying statements in WITH
--
-- INSERT ... RETURNING
WITH t AS (
INSERT INTO y
VALUES
(11),
(12),
(13),
(14),
(15),
(16),
(17),
(18),
(19),
(20)
RETURNING *
)
SELECT * FROM t;
a
----
11
12
13
14
15
16
17
18
19
20
(10 rows)
SELECT * FROM y;
a
----
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
(20 rows)
-- UPDATE ... RETURNING
WITH t AS (
UPDATE y
SET a=a+1
RETURNING *
)
SELECT * FROM t;
a
----
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
(20 rows)
SELECT * FROM y;
a
----
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
(20 rows)
-- DELETE ... RETURNING
WITH t AS (
DELETE FROM y
WHERE a <= 10
RETURNING *
)
SELECT * FROM t;
a
----
2
3
4
5
6
7
8
9
10
(9 rows)
SELECT * FROM y;
a
----
11
12
13
14
15
16
17
18
19
20
21
(11 rows)
-- forward reference
WITH RECURSIVE t AS (
INSERT INTO y
SELECT a+5 FROM t2 WHERE a > 5
RETURNING *
), t2 AS (
UPDATE y SET a=a-11 RETURNING *
)
SELECT * FROM t
UNION ALL
SELECT * FROM t2;
a
----
11
12
13
14
15
0
1
2
3
4
5
6
7
8
9
10
(16 rows)
SELECT * FROM y;
a
----
0
1
2
3
4
5
6
11
7
12
8
13
9
14
10
15
(16 rows)
-- unconditional DO INSTEAD rule
CREATE RULE y_rule AS ON DELETE TO y DO INSTEAD
INSERT INTO y VALUES(42) RETURNING *;
WITH t AS (
DELETE FROM y RETURNING *
)
SELECT * FROM t;
a
----
42
(1 row)
SELECT * FROM y;
a
----
0
1
2
3
4
5
6
11
7
12
8
13
9
14
10
15
42
(17 rows)
DROP RULE y_rule ON y;
-- check merging of outer CTE with CTE in a rule action
CREATE TEMP TABLE bug6051 AS
select i from generate_series(1,3) as t(i);
SELECT * FROM bug6051;
i
---
1
2
3
(3 rows)
WITH t1 AS ( DELETE FROM bug6051 RETURNING * )
INSERT INTO bug6051 SELECT * FROM t1;
SELECT * FROM bug6051;
i
---
1
2
3
(3 rows)
CREATE TEMP TABLE bug6051_2 (i int);
CREATE RULE bug6051_ins AS ON INSERT TO bug6051 DO INSTEAD
INSERT INTO bug6051_2
SELECT NEW.i;
WITH t1 AS ( DELETE FROM bug6051 RETURNING * )
INSERT INTO bug6051 SELECT * FROM t1;
SELECT * FROM bug6051;
i
---
(0 rows)
SELECT * FROM bug6051_2;
i
---
1
2
3
(3 rows)
-- a truly recursive CTE in the same list
WITH RECURSIVE t(a) AS (
SELECT 0
UNION ALL
SELECT a+1 FROM t WHERE a+1 < 5
), t2 as (
INSERT INTO y
SELECT * FROM t RETURNING *
)
SELECT * FROM t2 JOIN y USING (a) ORDER BY a;
a
---
0
1
2
3
4
(5 rows)
SELECT * FROM y;
a
----
0
1
2
3
4
5
6
11
7
12
8
13
9
14
10
15
42
0
1
2
3
4
(22 rows)
-- data-modifying WITH in a modifying statement
WITH t AS (
DELETE FROM y
WHERE a <= 10
RETURNING *
)
INSERT INTO y SELECT -a FROM t RETURNING *;
a
-----
0
-1
-2
-3
-4
-5
-6
-7
-8
-9
-10
0
-1
-2
-3
-4
(16 rows)
SELECT * FROM y;
a
-----
11
12
13
14
15
42
0
-1
-2
-3
-4
-5
-6
-7
-8
-9
-10
0
-1
-2
-3
-4
(22 rows)
-- check that WITH query is run to completion even if outer query isn't
WITH t AS (
UPDATE y SET a = a * 100 RETURNING *
)
SELECT * FROM t LIMIT 10;
a
------
1100
1200
1300
1400
1500
4200
0
-100
-200
-300
(10 rows)
SELECT * FROM y;
a
-------
1100
1200
1300
1400
1500
4200
0
-100
-200
-300
-400
-500
-600
-700
-800
-900
-1000
0
-100
-200
-300
-400
(22 rows)
-- data-modifying WITH containing INSERT...ON CONFLICT DO UPDATE
CREATE TABLE withz AS SELECT i AS k, (i || ' v')::text v FROM generate_series(1, 16, 3) i;
ALTER TABLE withz ADD UNIQUE (k);
WITH t AS (
INSERT INTO withz SELECT i, 'insert'
FROM generate_series(0, 16) i
ON CONFLICT (k) DO UPDATE SET v = withz.v || ', now update'
RETURNING *
)
SELECT * FROM t JOIN y ON t.k = y.a ORDER BY a, k;
k | v | a
---+--------+---
0 | insert | 0
0 | insert | 0
(2 rows)
-- Test EXCLUDED.* reference within CTE
WITH aa AS (
INSERT INTO withz VALUES(1, 5) ON CONFLICT (k) DO UPDATE SET v = EXCLUDED.v
WHERE withz.k != EXCLUDED.k
RETURNING *
)
SELECT * FROM aa;
k | v
---+---
(0 rows)
-- New query/snapshot demonstrates side-effects of previous query.
SELECT * FROM withz ORDER BY k;
k | v
----+------------------
0 | insert
1 | 1 v, now update
2 | insert
3 | insert
4 | 4 v, now update
5 | insert
6 | insert
7 | 7 v, now update
8 | insert
9 | insert
10 | 10 v, now update
11 | insert
12 | insert
13 | 13 v, now update
14 | insert
15 | insert
16 | 16 v, now update
(17 rows)
--
-- Ensure subqueries within the update clause work, even if they
-- reference outside values
--
WITH aa AS (SELECT 1 a, 2 b)
INSERT INTO withz VALUES(1, 'insert')
ON CONFLICT (k) DO UPDATE SET v = (SELECT b || ' update' FROM aa WHERE a = 1 LIMIT 1);
WITH aa AS (SELECT 1 a, 2 b)
INSERT INTO withz VALUES(1, 'insert')
ON CONFLICT (k) DO UPDATE SET v = ' update' WHERE withz.k = (SELECT a FROM aa);
WITH aa AS (SELECT 1 a, 2 b)
INSERT INTO withz VALUES(1, 'insert')
ON CONFLICT (k) DO UPDATE SET v = (SELECT b || ' update' FROM aa WHERE a = 1 LIMIT 1);
WITH aa AS (SELECT 'a' a, 'b' b UNION ALL SELECT 'a' a, 'b' b)
INSERT INTO withz VALUES(1, 'insert')
ON CONFLICT (k) DO UPDATE SET v = (SELECT b || ' update' FROM aa WHERE a = 'a' LIMIT 1);
WITH aa AS (SELECT 1 a, 2 b)
INSERT INTO withz VALUES(1, (SELECT b || ' insert' FROM aa WHERE a = 1 ))
ON CONFLICT (k) DO UPDATE SET v = (SELECT b || ' update' FROM aa WHERE a = 1 LIMIT 1);
-- Update a row more than once, in different parts of a wCTE. That is
-- an allowed, presumably very rare, edge case, but since it was
-- broken in the past, having a test seems worthwhile.
WITH simpletup AS (
SELECT 2 k, 'Green' v),
upsert_cte AS (
INSERT INTO withz VALUES(2, 'Blue') ON CONFLICT (k) DO
UPDATE SET (k, v) = (SELECT k, v FROM simpletup WHERE simpletup.k = withz.k)
RETURNING k, v)
INSERT INTO withz VALUES(2, 'Red') ON CONFLICT (k) DO
UPDATE SET (k, v) = (SELECT k, v FROM upsert_cte WHERE upsert_cte.k = withz.k)
RETURNING k, v;
k | v
---+---
(0 rows)
DROP TABLE withz;
-- check that run to completion happens in proper ordering
TRUNCATE TABLE y;
INSERT INTO y SELECT generate_series(1, 3);
CREATE TEMPORARY TABLE yy (a INTEGER);
WITH RECURSIVE t1 AS (
INSERT INTO y SELECT * FROM y RETURNING *
), t2 AS (
INSERT INTO yy SELECT * FROM t1 RETURNING *
)
SELECT 1;
?column?
----------
1
(1 row)
SELECT * FROM y;
a
---
1
2
3
1
2
3
(6 rows)
SELECT * FROM yy;
a
---
1
2
3
(3 rows)
WITH RECURSIVE t1 AS (
INSERT INTO yy SELECT * FROM t2 RETURNING *
), t2 AS (
INSERT INTO y SELECT * FROM y RETURNING *
)
SELECT 1;
?column?
----------
1
(1 row)
SELECT * FROM y;
a
---
1
2
3
1
2
3
1
2
3
1
2
3
(12 rows)
SELECT * FROM yy;
a
---
1
2
3
1
2
3
1
2
3
(9 rows)
-- triggers
TRUNCATE TABLE y;
INSERT INTO y SELECT generate_series(1, 10);
CREATE FUNCTION y_trigger() RETURNS trigger AS $$
begin
raise notice 'y_trigger: a = %', new.a;
return new;
end;
$$ LANGUAGE plpgsql;
CREATE TRIGGER y_trig BEFORE INSERT ON y FOR EACH ROW
EXECUTE PROCEDURE y_trigger();
WITH t AS (
INSERT INTO y
VALUES
(21),
(22),
(23)
RETURNING *
)
SELECT * FROM t;
NOTICE: y_trigger: a = 21
NOTICE: y_trigger: a = 22
NOTICE: y_trigger: a = 23
a
----
21
22
23
(3 rows)
SELECT * FROM y;
a
----
1
2
3
4
5
6
7
8
9
10
21
22
23
(13 rows)
DROP TRIGGER y_trig ON y;
CREATE TRIGGER y_trig AFTER INSERT ON y FOR EACH ROW
EXECUTE PROCEDURE y_trigger();
WITH t AS (
INSERT INTO y
VALUES
(31),
(32),
(33)
RETURNING *
)
SELECT * FROM t LIMIT 1;
NOTICE: y_trigger: a = 31
NOTICE: y_trigger: a = 32
NOTICE: y_trigger: a = 33
a
----
31
(1 row)
SELECT * FROM y;
a
----
1
2
3
4
5
6
7
8
9
10
21
22
23
31
32
33
(16 rows)
DROP TRIGGER y_trig ON y;
CREATE OR REPLACE FUNCTION y_trigger() RETURNS trigger AS $$
begin
raise notice 'y_trigger';
return null;
end;
$$ LANGUAGE plpgsql;
CREATE TRIGGER y_trig AFTER INSERT ON y FOR EACH STATEMENT
EXECUTE PROCEDURE y_trigger();
WITH t AS (
INSERT INTO y
VALUES
(41),
(42),
(43)
RETURNING *
)
SELECT * FROM t;
NOTICE: y_trigger
a
----
41
42
43
(3 rows)
SELECT * FROM y;
a
----
1
2
3
4
5
6
7
8
9
10
21
22
23
31
32
33
41
42
43
(19 rows)
DROP TRIGGER y_trig ON y;
DROP FUNCTION y_trigger();
-- WITH attached to inherited UPDATE or DELETE
CREATE TEMP TABLE parent ( id int, val text );
CREATE TEMP TABLE child1 ( ) INHERITS ( parent );
CREATE TEMP TABLE child2 ( ) INHERITS ( parent );
INSERT INTO parent VALUES ( 1, 'p1' );
INSERT INTO child1 VALUES ( 11, 'c11' ),( 12, 'c12' );
INSERT INTO child2 VALUES ( 23, 'c21' ),( 24, 'c22' );
WITH rcte AS ( SELECT sum(id) AS totalid FROM parent )
UPDATE parent SET id = id + totalid FROM rcte;
SELECT * FROM parent;
id | val
----+-----
72 | p1
82 | c11
83 | c12
94 | c21
95 | c22
(5 rows)
WITH wcte AS ( INSERT INTO child1 VALUES ( 42, 'new' ) RETURNING id AS newid )
UPDATE parent SET id = id + newid FROM wcte;
SELECT * FROM parent;
id | val
-----+-----
114 | p1
42 | new
124 | c11
125 | c12
136 | c21
137 | c22
(6 rows)
WITH rcte AS ( SELECT max(id) AS maxid FROM parent )
DELETE FROM parent USING rcte WHERE id = maxid;
SELECT * FROM parent;
id | val
-----+-----
114 | p1
42 | new
124 | c11
125 | c12
136 | c21
(5 rows)
WITH wcte AS ( INSERT INTO child2 VALUES ( 42, 'new2' ) RETURNING id AS newid )
DELETE FROM parent USING wcte WHERE id = newid;
SELECT * FROM parent;
id | val
-----+------
114 | p1
124 | c11
125 | c12
136 | c21
42 | new2
(5 rows)
-- check EXPLAIN VERBOSE for a wCTE with RETURNING
EXPLAIN (VERBOSE, COSTS OFF)
WITH wcte AS ( INSERT INTO int8_tbl VALUES ( 42, 47 ) RETURNING q2 )
DELETE FROM a USING wcte WHERE aa = q2;
QUERY PLAN
----------------------------------------------------
Delete on public.a
Delete on public.a
Delete on public.b
Delete on public.c
Delete on public.d
CTE wcte
-> Insert on public.int8_tbl
Output: int8_tbl.q2
-> Result
Output: '42'::bigint, '47'::bigint
-> Nested Loop
Output: a.ctid, wcte.*
Join Filter: (a.aa = wcte.q2)
-> Seq Scan on public.a
Output: a.ctid, a.aa
-> CTE Scan on wcte
Output: wcte.*, wcte.q2
-> Nested Loop
Output: b.ctid, wcte.*
Join Filter: (b.aa = wcte.q2)
-> Seq Scan on public.b
Output: b.ctid, b.aa
-> CTE Scan on wcte
Output: wcte.*, wcte.q2
-> Nested Loop
Output: c.ctid, wcte.*
Join Filter: (c.aa = wcte.q2)
-> Seq Scan on public.c
Output: c.ctid, c.aa
-> CTE Scan on wcte
Output: wcte.*, wcte.q2
-> Nested Loop
Output: d.ctid, wcte.*
Join Filter: (d.aa = wcte.q2)
-> Seq Scan on public.d
Output: d.ctid, d.aa
-> CTE Scan on wcte
Output: wcte.*, wcte.q2
(38 rows)
-- error cases
-- data-modifying WITH tries to use its own output
WITH RECURSIVE t AS (
INSERT INTO y
SELECT * FROM t
)
VALUES(FALSE);
ERROR: recursive query "t" must not contain data-modifying statements
LINE 1: WITH RECURSIVE t AS (
^
-- no RETURNING in a referenced data-modifying WITH
WITH t AS (
INSERT INTO y VALUES(0)
)
SELECT * FROM t;
ERROR: WITH query "t" does not have a RETURNING clause
LINE 4: SELECT * FROM t;
^
-- data-modifying WITH allowed only at the top level
SELECT * FROM (
WITH t AS (UPDATE y SET a=a+1 RETURNING *)
SELECT * FROM t
) ss;
ERROR: WITH clause containing a data-modifying statement must be at the top level
LINE 2: WITH t AS (UPDATE y SET a=a+1 RETURNING *)
^
-- most variants of rules aren't allowed
CREATE RULE y_rule AS ON INSERT TO y WHERE a=0 DO INSTEAD DELETE FROM y;
WITH t AS (
INSERT INTO y VALUES(0)
)
VALUES(FALSE);
ERROR: conditional DO INSTEAD rules are not supported for data-modifying statements in WITH
DROP RULE y_rule ON y;
-- check that parser lookahead for WITH doesn't cause any odd behavior
create table foo (with baz); -- fail, WITH is a reserved word
ERROR: syntax error at or near "with"
LINE 1: create table foo (with baz);
^
create table foo (with ordinality); -- fail, WITH is a reserved word
ERROR: syntax error at or near "with"
LINE 1: create table foo (with ordinality);
^
with ordinality as (select 1 as x) select * from ordinality;
x
---
1
(1 row)
-- check sane response to attempt to modify CTE relation
WITH test AS (SELECT 42) INSERT INTO test VALUES (1);
ERROR: relation "test" does not exist
LINE 1: WITH test AS (SELECT 42) INSERT INTO test VALUES (1);
^
-- check response to attempt to modify table with same name as a CTE (perhaps
-- surprisingly it works, because CTEs don't hide tables from data-modifying
-- statements)
create temp table test (i int);
with test as (select 42) insert into test select * from test;
select * from test;
i
----
42
(1 row)
drop table test;
[text/plain] window.out (116.5K, ../../20180725054236.GB17411@fetter.org/5-window.out)
download
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
@ 2018-07-25 15:18 ` Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
1 sibling, 1 reply; 112+ messages in thread
From: Andrew Gierth @ 2018-07-25 15:18 UTC (permalink / raw)
To: David Fetter <david@fetter.org>; +Cc: pgsql-hackers
>>>>> "David" == David Fetter <david@fetter.org> writes:
David> Please find attached a version rebased atop 167075be3ab1547e18
David> with what I believe are appropriate changes to regression test
David> output. The other changes to the regression tests output are
David> somewhat puzzling, as they change the actual results of queries.
Both of those changes are the result of volatile CTEs being inlined more
than once (in one case, as part of an explicit test to ensure that CTEs
are being materialized and not multiply evaluated).
If you look for the XXX comment in the patch, it should be easy to add a
check that skips inlining if cterefcount > 1 and
contains_volatile_functions is true.
--
Andrew (irc:RhodiumToad)
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
@ 2018-07-25 19:14 ` David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
0 siblings, 1 reply; 112+ messages in thread
From: David Fetter @ 2018-07-25 19:14 UTC (permalink / raw)
To: Andrew Gierth <andrew@tao11.riddles.org.uk>; +Cc: pgsql-hackers
On Wed, Jul 25, 2018 at 04:18:42PM +0100, Andrew Gierth wrote:
> >>>>> "David" == David Fetter <david@fetter.org> writes:
>
> David> Please find attached a version rebased atop 167075be3ab1547e18
> David> with what I believe are appropriate changes to regression test
> David> output. The other changes to the regression tests output are
> David> somewhat puzzling, as they change the actual results of queries.
>
> Both of those changes are the result of volatile CTEs being inlined more
> than once (in one case, as part of an explicit test to ensure that CTEs
> are being materialized and not multiply evaluated).
>
> If you look for the XXX comment in the patch, it should be easy to add a
> check that skips inlining if cterefcount > 1 and
> contains_volatile_functions is true.
Thanks for the broad hints!
Please find attached the next version, which passes 'make check'.
Best,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778
Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate
Attachments:
[text/x-diff] 0001-Inlining-CTEs-v0003.patch (16.0K, ../../20180725191447.GE17411@fetter.org/2-0001-Inlining-CTEs-v0003.patch)
download | inline diff:
From 92cc38740341b74dd7d26c0fd80d285554faf038 Mon Sep 17 00:00:00 2001
From: David Fetter <david@fetter.org>
Date: Tue, 24 Jul 2018 22:09:55 -0700
Subject: [PATCH] Inlining CTEs v0003
To: pgsql-hackers@postgresql.org
---
src/backend/nodes/copyfuncs.c | 1 +
src/backend/nodes/equalfuncs.c | 1 +
src/backend/nodes/outfuncs.c | 1 +
src/backend/nodes/readfuncs.c | 1 +
src/backend/optimizer/plan/subselect.c | 152 ++++++++++++++++++++++
src/include/nodes/parsenodes.h | 1 +
src/test/regress/expected/rowsecurity.out | 92 ++++++-------
src/test/regress/expected/rowtypes.out | 13 +-
src/test/regress/expected/rules.out | 6 +-
9 files changed, 201 insertions(+), 67 deletions(-)
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 17b650b8cb..8ba2fc1aed 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2530,6 +2530,7 @@ _copyCommonTableExpr(const CommonTableExpr *from)
COPY_NODE_FIELD(ctequery);
COPY_LOCATION_FIELD(location);
COPY_SCALAR_FIELD(cterecursive);
+ COPY_SCALAR_FIELD(ctematerialized);
COPY_SCALAR_FIELD(cterefcount);
COPY_NODE_FIELD(ctecolnames);
COPY_NODE_FIELD(ctecoltypes);
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index 378f2facb8..2a04506d0d 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2793,6 +2793,7 @@ _equalCommonTableExpr(const CommonTableExpr *a, const CommonTableExpr *b)
COMPARE_NODE_FIELD(ctequery);
COMPARE_LOCATION_FIELD(location);
COMPARE_SCALAR_FIELD(cterecursive);
+ COMPARE_SCALAR_FIELD(ctematerialized);
COMPARE_SCALAR_FIELD(cterefcount);
COMPARE_NODE_FIELD(ctecolnames);
COMPARE_NODE_FIELD(ctecoltypes);
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index a6454ce28b..0ea6b7125a 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -3080,6 +3080,7 @@ _outCommonTableExpr(StringInfo str, const CommonTableExpr *node)
WRITE_NODE_FIELD(ctequery);
WRITE_LOCATION_FIELD(location);
WRITE_BOOL_FIELD(cterecursive);
+ WRITE_BOOL_FIELD(ctematerialized);
WRITE_INT_FIELD(cterefcount);
WRITE_NODE_FIELD(ctecolnames);
WRITE_NODE_FIELD(ctecoltypes);
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index 9a01eb6b63..01db3c1c85 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -409,6 +409,7 @@ _readCommonTableExpr(void)
READ_NODE_FIELD(ctequery);
READ_LOCATION_FIELD(location);
READ_BOOL_FIELD(cterecursive);
+ READ_BOOL_FIELD(ctematerialized);
READ_INT_FIELD(cterefcount);
READ_NODE_FIELD(ctecolnames);
READ_NODE_FIELD(ctecoltypes);
diff --git a/src/backend/optimizer/plan/subselect.c b/src/backend/optimizer/plan/subselect.c
index 83008d7661..5aa242ff26 100644
--- a/src/backend/optimizer/plan/subselect.c
+++ b/src/backend/optimizer/plan/subselect.c
@@ -1146,6 +1146,141 @@ hash_ok_operator(OpExpr *expr)
}
+struct inline_cte_walker_ctx
+{
+ const char *ctename;
+ int levelsup;
+ int refcount;
+ Query *ctequery;
+ CommonTableExpr *cte;
+};
+
+static bool inline_cte_walker(Node *node, void *ctxp)
+{
+ struct inline_cte_walker_ctx *ctx = ctxp;
+
+ if (!node)
+ return false;
+
+ if (IsA(node, Query))
+ {
+ /*
+ * This one is a bit tricky. It's our job to handle the recursion here,
+ * but we do some of the lifting normally handled by query_tree_walker
+ * in order to get the sequence of operations right.
+ *
+ * First, if the Query we're looking at is the one containing our CTE
+ * definition, then we don't need to recurse into our own CTE or CTEs
+ * that are earlier in the list than ours (since cteList has been
+ * sorted for us into dependency order). We could check whether a
+ * nested query is hiding ours, but that seems too much of an edge case
+ * to be worth optimizing (the levelsup check will ensure we don't
+ * replace any CTEs improperly). So we scan the cteList ourselves
+ * rather than having query_tree_walker do it.
+ *
+ * Second, we want to walk the rangetable _before_ replacing any
+ * RTE_CTE nodes, in order to avoid re-walking the subquery we just
+ * inlined. (range_table_walker, if told to visit the RTE nodes at all,
+ * visits them before their content.) So we have range_table_walker
+ * ignore the RTE nodes themselves and only walk their contents.
+ *
+ * Third, we scan the rangetable for RTE_CTE nodes to replace.
+ *
+ * Fourth, we use query_tree_walker to find and walk the rest of the
+ * query, telling it to ignore the rangetable and CTEs.
+ *
+ * Note that ctx->levelsup is -1 on entry the first time, since we need
+ * the incremented value to be 0 when scanning the content of the query
+ * containing the definition.
+ */
+ Query *query = castNode(Query, node);
+ ListCell *lc;
+ bool do_replace = ctx->levelsup >= 0;
+
+ ctx->levelsup++;
+
+ foreach (lc, query->cteList)
+ {
+ CommonTableExpr *cte = lfirst_node(CommonTableExpr, lc);
+
+ if (!do_replace && strcmp(cte->ctename, ctx->ctename) == 0)
+ do_replace = true;
+ else if (do_replace)
+ inline_cte_walker(cte->ctequery, ctxp);
+ }
+
+ range_table_walker(query->rtable, inline_cte_walker, ctxp, 0);
+
+ foreach (lc, query->rtable)
+ {
+ RangeTblEntry *rte = lfirst_node(RangeTblEntry, lc);
+
+ if (rte->rtekind == RTE_CTE &&
+ strcmp(rte->ctename, ctx->ctename) == 0 &&
+ rte->ctelevelsup == ctx->levelsup)
+ {
+ Query *newquery = ctx->ctequery;
+
+ /*
+ * We need to do some work here that view rewrite does not, and
+ * in turn we do not do some work that view rewrite does.
+ *
+ * Firstly, views can't have outer references but CTEs can
+ * (especially in the case of CTEs referencing other CTEs), so
+ * we need to fix up all levelsup attributes inside the CTE
+ * query.
+ *
+ * Secondly, views (and explicit subqueries) currently have
+ * different behaviour w.r.t. SELECT FOR UPDATE than CTEs do. A
+ * FOR UPDATE clause is treated as extending into views and
+ * subqueries, but not into CTEs. We preserve this distinction
+ * by not trying to push rowmarks into the new subquery.
+ *
+ * We avoid copyObject if possible because subquery processing
+ * copies the query too.
+ */
+ if (ctx->levelsup > 0)
+ {
+ newquery = copyObject(newquery);
+ IncrementVarSublevelsUp((Node *) newquery, ctx->levelsup, 1);
+ }
+
+ /*
+ * Here's where we do the actual substitution.
+ */
+ rte->rtekind = RTE_SUBQUERY;
+ rte->subquery = newquery;
+ rte->security_barrier = false;
+
+ ctx->refcount--;
+ }
+ }
+
+ query_tree_walker(query, inline_cte_walker, ctxp,
+ QTW_IGNORE_RANGE_TABLE | QTW_IGNORE_CTE_SUBQUERIES);
+
+ ctx->levelsup--;
+
+ return false;
+ }
+
+ return expression_tree_walker(node, inline_cte_walker, ctxp);
+}
+
+static void inline_cte(PlannerInfo *root, CommonTableExpr *cte)
+{
+ struct inline_cte_walker_ctx ctx;
+ ctx.ctequery = castNode(Query, cte->ctequery);
+ ctx.ctename = cte->ctename;
+ ctx.refcount = cte->cterefcount;
+ ctx.levelsup = -1;
+ ctx.cte = cte;
+ inline_cte_walker((Node *) root->parse, &ctx);
+ /* we must replace all references */
+ Assert(ctx.refcount == 0);
+}
+
+
/*
* SS_process_ctes: process a query's WITH list
*
@@ -1183,6 +1318,23 @@ SS_process_ctes(PlannerInfo *root)
continue;
}
+ /*
+ * Consider inlining the CTE rather than planning it separately.
+ */
+ if (cmdType == CMD_SELECT &&
+ !cte->ctematerialized &&
+ !cte->cterecursive &&
+ !((cte->cterefcount > 1) && contain_volatile_functions(cte->ctequery)) &&
+ (castNode(Query, cte->ctequery)->rowMarks == NIL)
+ )
+ {
+ inline_cte(root, cte);
+
+ /* Make a dummy entry in cte_plan_ids */
+ root->cte_plan_ids = lappend_int(root->cte_plan_ids, -1);
+ continue;
+ }
+
/*
* Copy the source Query node. Probably not necessary, but let's keep
* this similar to make_subplan.
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 7855cff30d..047d3de71c 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -1397,6 +1397,7 @@ typedef struct CommonTableExpr
int location; /* token location, or -1 if unknown */
/* These fields are set during parse analysis: */
bool cterecursive; /* is this CTE actually recursive? */
+ bool ctematerialized; /* is this an optimization fence? */
int cterefcount; /* number of RTEs referencing this CTE
* (excluding internal self-references) */
List *ctecolnames; /* list of output column names */
diff --git a/src/test/regress/expected/rowsecurity.out b/src/test/regress/expected/rowsecurity.out
index bc16ca4c43..0d8d382830 100644
--- a/src/test/regress/expected/rowsecurity.out
+++ b/src/test/regress/expected/rowsecurity.out
@@ -2181,29 +2181,25 @@ EXPLAIN (COSTS OFF) EXECUTE plancache_test;
PREPARE plancache_test2 AS WITH q AS (SELECT * FROM z1 WHERE f_leak(b)) SELECT * FROM q,z2;
EXPLAIN (COSTS OFF) EXECUTE plancache_test2;
- QUERY PLAN
--------------------------------------------------
+ QUERY PLAN
+-----------------------------------------------------
Nested Loop
- CTE q
- -> Seq Scan on z1
- Filter: (((a % 2) = 0) AND f_leak(b))
- -> CTE Scan on q
+ -> Seq Scan on z2
-> Materialize
- -> Seq Scan on z2
-(7 rows)
+ -> Seq Scan on z1
+ Filter: (((a % 2) = 0) AND f_leak(b))
+(5 rows)
PREPARE plancache_test3 AS WITH q AS (SELECT * FROM z2) SELECT * FROM q,z1 WHERE f_leak(z1.b);
EXPLAIN (COSTS OFF) EXECUTE plancache_test3;
QUERY PLAN
-----------------------------------------------------
Nested Loop
- CTE q
- -> Seq Scan on z2
- -> CTE Scan on q
+ -> Seq Scan on z2
-> Materialize
-> Seq Scan on z1
Filter: (((a % 2) = 0) AND f_leak(b))
-(7 rows)
+(5 rows)
SET ROLE regress_rls_group1;
SELECT * FROM z1 WHERE f_leak(b);
@@ -2230,28 +2226,24 @@ EXPLAIN (COSTS OFF) EXECUTE plancache_test;
(2 rows)
EXPLAIN (COSTS OFF) EXECUTE plancache_test2;
- QUERY PLAN
--------------------------------------------------
+ QUERY PLAN
+-----------------------------------------------------
Nested Loop
- CTE q
- -> Seq Scan on z1
- Filter: (((a % 2) = 0) AND f_leak(b))
- -> CTE Scan on q
+ -> Seq Scan on z2
-> Materialize
- -> Seq Scan on z2
-(7 rows)
+ -> Seq Scan on z1
+ Filter: (((a % 2) = 0) AND f_leak(b))
+(5 rows)
EXPLAIN (COSTS OFF) EXECUTE plancache_test3;
QUERY PLAN
-----------------------------------------------------
Nested Loop
- CTE q
- -> Seq Scan on z2
- -> CTE Scan on q
+ -> Seq Scan on z2
-> Materialize
-> Seq Scan on z1
Filter: (((a % 2) = 0) AND f_leak(b))
-(7 rows)
+(5 rows)
SET SESSION AUTHORIZATION regress_rls_carol;
SELECT * FROM z1 WHERE f_leak(b);
@@ -2278,28 +2270,24 @@ EXPLAIN (COSTS OFF) EXECUTE plancache_test;
(2 rows)
EXPLAIN (COSTS OFF) EXECUTE plancache_test2;
- QUERY PLAN
--------------------------------------------------
+ QUERY PLAN
+-----------------------------------------------------
Nested Loop
- CTE q
- -> Seq Scan on z1
- Filter: (((a % 2) = 1) AND f_leak(b))
- -> CTE Scan on q
+ -> Seq Scan on z2
-> Materialize
- -> Seq Scan on z2
-(7 rows)
+ -> Seq Scan on z1
+ Filter: (((a % 2) = 1) AND f_leak(b))
+(5 rows)
EXPLAIN (COSTS OFF) EXECUTE plancache_test3;
QUERY PLAN
-----------------------------------------------------
Nested Loop
- CTE q
- -> Seq Scan on z2
- -> CTE Scan on q
+ -> Seq Scan on z2
-> Materialize
-> Seq Scan on z1
Filter: (((a % 2) = 1) AND f_leak(b))
-(7 rows)
+(5 rows)
SET ROLE regress_rls_group2;
SELECT * FROM z1 WHERE f_leak(b);
@@ -2326,28 +2314,24 @@ EXPLAIN (COSTS OFF) EXECUTE plancache_test;
(2 rows)
EXPLAIN (COSTS OFF) EXECUTE plancache_test2;
- QUERY PLAN
--------------------------------------------------
+ QUERY PLAN
+-----------------------------------------------------
Nested Loop
- CTE q
- -> Seq Scan on z1
- Filter: (((a % 2) = 1) AND f_leak(b))
- -> CTE Scan on q
+ -> Seq Scan on z2
-> Materialize
- -> Seq Scan on z2
-(7 rows)
+ -> Seq Scan on z1
+ Filter: (((a % 2) = 1) AND f_leak(b))
+(5 rows)
EXPLAIN (COSTS OFF) EXECUTE plancache_test3;
QUERY PLAN
-----------------------------------------------------
Nested Loop
- CTE q
- -> Seq Scan on z2
- -> CTE Scan on q
+ -> Seq Scan on z2
-> Materialize
-> Seq Scan on z1
Filter: (((a % 2) = 1) AND f_leak(b))
-(7 rows)
+(5 rows)
--
-- Views should follow policy for view owner.
@@ -2854,13 +2838,11 @@ NOTICE: f_leak => 98f13708210194c475687be6106a3b84
(11 rows)
EXPLAIN (COSTS OFF) WITH cte1 AS (SELECT * FROM t1 WHERE f_leak(b)) SELECT * FROM cte1;
- QUERY PLAN
--------------------------------------------------
- CTE Scan on cte1
- CTE cte1
- -> Seq Scan on t1
- Filter: (((a % 2) = 0) AND f_leak(b))
-(4 rows)
+ QUERY PLAN
+-----------------------------------------
+ Seq Scan on t1
+ Filter: (((a % 2) = 0) AND f_leak(b))
+(2 rows)
WITH cte1 AS (UPDATE t1 SET a = a + 1 RETURNING *) SELECT * FROM cte1; --fail
ERROR: new row violates row-level security policy for table "t1"
diff --git a/src/test/regress/expected/rowtypes.out b/src/test/regress/expected/rowtypes.out
index 30053d07df..c0df2f6333 100644
--- a/src/test/regress/expected/rowtypes.out
+++ b/src/test/regress/expected/rowtypes.out
@@ -1041,14 +1041,11 @@ with r(a,b) as
(values (1,row(1,2)), (1,row(null,null)), (1,null),
(null,row(1,2)), (null,row(null,null)), (null,null) )
select r, r is null as isnull, r is not null as isnotnull from r;
- QUERY PLAN
-----------------------------------------------------------
- CTE Scan on r
- Output: r.*, (r.* IS NULL), (r.* IS NOT NULL)
- CTE r
- -> Values Scan on "*VALUES*"
- Output: "*VALUES*".column1, "*VALUES*".column2
-(5 rows)
+ QUERY PLAN
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ Values Scan on "*VALUES*"
+ Output: ROW("*VALUES*".column1, "*VALUES*".column2), (("*VALUES*".column1 IS NULL) AND ("*VALUES*".column2 IS NOT DISTINCT FROM NULL)), (("*VALUES*".column1 IS NOT NULL) AND ("*VALUES*".column2 IS DISTINCT FROM NULL))
+(2 rows)
with r(a,b) as
(values (1,row(1,2)), (1,row(null,null)), (1,null),
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 744d501e31..882469b79f 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -3135,10 +3135,8 @@ RETURNING *;
Conflict Resolution: UPDATE
Conflict Arbiter Indexes: hat_data_unique_idx
Conflict Filter: ((excluded.hat_color <> 'forbidden'::bpchar) AND (hat_data.* <> excluded.*))
- CTE data
- -> Values Scan on "*VALUES*"
- -> CTE Scan on data
-(7 rows)
+ -> Values Scan on "*VALUES*"
+(5 rows)
SELECT * FROM hat_data WHERE hat_name IN ('h8', 'h9', 'h7') ORDER BY hat_name;
hat_name | hat_color
--
2.18.0
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
@ 2018-07-27 02:55 ` Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
0 siblings, 1 reply; 112+ messages in thread
From: Thomas Munro @ 2018-07-27 02:55 UTC (permalink / raw)
To: David Fetter <david@fetter.org>; +Cc: Andrew Gierth <andrew@tao11.riddles.org.uk>; pgsql-hackers
On Thu, Jul 26, 2018 at 7:14 AM, David Fetter <david@fetter.org> wrote:
> Please find attached the next version, which passes 'make check'.
... but not 'make check-world' (contrib/postgres_fdw's EXPLAIN is different).
--
Thomas Munro
http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
@ 2018-07-27 08:10 ` David Fetter <david@fetter.org>
2018-08-08 04:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
0 siblings, 2 replies; 112+ messages in thread
From: David Fetter @ 2018-07-27 08:10 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@enterprisedb.com>; +Cc: Andrew Gierth <andrew@tao11.riddles.org.uk>; pgsql-hackers
On Fri, Jul 27, 2018 at 02:55:26PM +1200, Thomas Munro wrote:
> On Thu, Jul 26, 2018 at 7:14 AM, David Fetter <david@fetter.org> wrote:
> > Please find attached the next version, which passes 'make check'.
>
> ... but not 'make check-world' (contrib/postgres_fdw's EXPLAIN is different).
Please find attached a patch that does.
It doesn't always pass make installcheck-world, but I need to sleep
rather than investigate that at the moment.
Best,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778
Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate
Attachments:
[text/x-diff] 0001-Inlining-CTEs-v0004.patch (18.2K, ../../20180727081058.GN17411@fetter.org/2-0001-Inlining-CTEs-v0004.patch)
download | inline diff:
From f2c2b80653056058b7fb799bb8b9d2c0443ddd5f Mon Sep 17 00:00:00 2001
From: David Fetter <david@fetter.org>
Date: Tue, 24 Jul 2018 22:09:55 -0700
Subject: [PATCH] Inlining CTEs v0004
To: pgsql-hackers@postgresql.org
---
.../postgres_fdw/expected/postgres_fdw.out | 22 +--
src/backend/nodes/copyfuncs.c | 1 +
src/backend/nodes/equalfuncs.c | 1 +
src/backend/nodes/outfuncs.c | 1 +
src/backend/nodes/readfuncs.c | 1 +
src/backend/optimizer/plan/subselect.c | 152 ++++++++++++++++++
src/include/nodes/parsenodes.h | 1 +
src/test/regress/expected/rowsecurity.out | 92 +++++------
src/test/regress/expected/rowtypes.out | 13 +-
src/test/regress/expected/rules.out | 6 +-
10 files changed, 209 insertions(+), 81 deletions(-)
diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out
index f5498c62bd..8bf011df46 100644
--- a/contrib/postgres_fdw/expected/postgres_fdw.out
+++ b/contrib/postgres_fdw/expected/postgres_fdw.out
@@ -1889,21 +1889,15 @@ SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) ORDER BY t1.c3, t
-- join in CTE
EXPLAIN (VERBOSE, COSTS OFF)
WITH t (c1_1, c1_3, c2_1) AS (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
- QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------
+ QUERY PLAN
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Limit
- Output: t.c1_1, t.c2_1, t.c1_3
- CTE t
- -> Foreign Scan
- Output: t1.c1, t1.c3, t2.c1
- Relations: (public.ft1 t1) INNER JOIN (public.ft2 t2)
- Remote SQL: SELECT r1."C 1", r1.c3, r2."C 1" FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r1."C 1" = r2."C 1"))))
- -> Sort
- Output: t.c1_1, t.c2_1, t.c1_3
- Sort Key: t.c1_3, t.c1_1
- -> CTE Scan on t
- Output: t.c1_1, t.c2_1, t.c1_3
-(12 rows)
+ Output: t1.c1, t2.c1, t1.c3
+ -> Foreign Scan
+ Output: t1.c1, t2.c1, t1.c3
+ Relations: (public.ft1 t1) INNER JOIN (public.ft2 t2)
+ Remote SQL: SELECT r2."C 1", r3."C 1", r2.c3 FROM ("S 1"."T 1" r2 INNER JOIN "S 1"."T 1" r3 ON (((r2."C 1" = r3."C 1")))) ORDER BY r2.c3 ASC NULLS LAST, r2."C 1" ASC NULLS LAST
+(6 rows)
WITH t (c1_1, c1_3, c2_1) AS (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
c1_1 | c2_1
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 17b650b8cb..8ba2fc1aed 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2530,6 +2530,7 @@ _copyCommonTableExpr(const CommonTableExpr *from)
COPY_NODE_FIELD(ctequery);
COPY_LOCATION_FIELD(location);
COPY_SCALAR_FIELD(cterecursive);
+ COPY_SCALAR_FIELD(ctematerialized);
COPY_SCALAR_FIELD(cterefcount);
COPY_NODE_FIELD(ctecolnames);
COPY_NODE_FIELD(ctecoltypes);
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index 378f2facb8..2a04506d0d 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2793,6 +2793,7 @@ _equalCommonTableExpr(const CommonTableExpr *a, const CommonTableExpr *b)
COMPARE_NODE_FIELD(ctequery);
COMPARE_LOCATION_FIELD(location);
COMPARE_SCALAR_FIELD(cterecursive);
+ COMPARE_SCALAR_FIELD(ctematerialized);
COMPARE_SCALAR_FIELD(cterefcount);
COMPARE_NODE_FIELD(ctecolnames);
COMPARE_NODE_FIELD(ctecoltypes);
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index a6454ce28b..0ea6b7125a 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -3080,6 +3080,7 @@ _outCommonTableExpr(StringInfo str, const CommonTableExpr *node)
WRITE_NODE_FIELD(ctequery);
WRITE_LOCATION_FIELD(location);
WRITE_BOOL_FIELD(cterecursive);
+ WRITE_BOOL_FIELD(ctematerialized);
WRITE_INT_FIELD(cterefcount);
WRITE_NODE_FIELD(ctecolnames);
WRITE_NODE_FIELD(ctecoltypes);
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index 9a01eb6b63..01db3c1c85 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -409,6 +409,7 @@ _readCommonTableExpr(void)
READ_NODE_FIELD(ctequery);
READ_LOCATION_FIELD(location);
READ_BOOL_FIELD(cterecursive);
+ READ_BOOL_FIELD(ctematerialized);
READ_INT_FIELD(cterefcount);
READ_NODE_FIELD(ctecolnames);
READ_NODE_FIELD(ctecoltypes);
diff --git a/src/backend/optimizer/plan/subselect.c b/src/backend/optimizer/plan/subselect.c
index 83008d7661..5aa242ff26 100644
--- a/src/backend/optimizer/plan/subselect.c
+++ b/src/backend/optimizer/plan/subselect.c
@@ -1146,6 +1146,141 @@ hash_ok_operator(OpExpr *expr)
}
+struct inline_cte_walker_ctx
+{
+ const char *ctename;
+ int levelsup;
+ int refcount;
+ Query *ctequery;
+ CommonTableExpr *cte;
+};
+
+static bool inline_cte_walker(Node *node, void *ctxp)
+{
+ struct inline_cte_walker_ctx *ctx = ctxp;
+
+ if (!node)
+ return false;
+
+ if (IsA(node, Query))
+ {
+ /*
+ * This one is a bit tricky. It's our job to handle the recursion here,
+ * but we do some of the lifting normally handled by query_tree_walker
+ * in order to get the sequence of operations right.
+ *
+ * First, if the Query we're looking at is the one containing our CTE
+ * definition, then we don't need to recurse into our own CTE or CTEs
+ * that are earlier in the list than ours (since cteList has been
+ * sorted for us into dependency order). We could check whether a
+ * nested query is hiding ours, but that seems too much of an edge case
+ * to be worth optimizing (the levelsup check will ensure we don't
+ * replace any CTEs improperly). So we scan the cteList ourselves
+ * rather than having query_tree_walker do it.
+ *
+ * Second, we want to walk the rangetable _before_ replacing any
+ * RTE_CTE nodes, in order to avoid re-walking the subquery we just
+ * inlined. (range_table_walker, if told to visit the RTE nodes at all,
+ * visits them before their content.) So we have range_table_walker
+ * ignore the RTE nodes themselves and only walk their contents.
+ *
+ * Third, we scan the rangetable for RTE_CTE nodes to replace.
+ *
+ * Fourth, we use query_tree_walker to find and walk the rest of the
+ * query, telling it to ignore the rangetable and CTEs.
+ *
+ * Note that ctx->levelsup is -1 on entry the first time, since we need
+ * the incremented value to be 0 when scanning the content of the query
+ * containing the definition.
+ */
+ Query *query = castNode(Query, node);
+ ListCell *lc;
+ bool do_replace = ctx->levelsup >= 0;
+
+ ctx->levelsup++;
+
+ foreach (lc, query->cteList)
+ {
+ CommonTableExpr *cte = lfirst_node(CommonTableExpr, lc);
+
+ if (!do_replace && strcmp(cte->ctename, ctx->ctename) == 0)
+ do_replace = true;
+ else if (do_replace)
+ inline_cte_walker(cte->ctequery, ctxp);
+ }
+
+ range_table_walker(query->rtable, inline_cte_walker, ctxp, 0);
+
+ foreach (lc, query->rtable)
+ {
+ RangeTblEntry *rte = lfirst_node(RangeTblEntry, lc);
+
+ if (rte->rtekind == RTE_CTE &&
+ strcmp(rte->ctename, ctx->ctename) == 0 &&
+ rte->ctelevelsup == ctx->levelsup)
+ {
+ Query *newquery = ctx->ctequery;
+
+ /*
+ * We need to do some work here that view rewrite does not, and
+ * in turn we do not do some work that view rewrite does.
+ *
+ * Firstly, views can't have outer references but CTEs can
+ * (especially in the case of CTEs referencing other CTEs), so
+ * we need to fix up all levelsup attributes inside the CTE
+ * query.
+ *
+ * Secondly, views (and explicit subqueries) currently have
+ * different behaviour w.r.t. SELECT FOR UPDATE than CTEs do. A
+ * FOR UPDATE clause is treated as extending into views and
+ * subqueries, but not into CTEs. We preserve this distinction
+ * by not trying to push rowmarks into the new subquery.
+ *
+ * We avoid copyObject if possible because subquery processing
+ * copies the query too.
+ */
+ if (ctx->levelsup > 0)
+ {
+ newquery = copyObject(newquery);
+ IncrementVarSublevelsUp((Node *) newquery, ctx->levelsup, 1);
+ }
+
+ /*
+ * Here's where we do the actual substitution.
+ */
+ rte->rtekind = RTE_SUBQUERY;
+ rte->subquery = newquery;
+ rte->security_barrier = false;
+
+ ctx->refcount--;
+ }
+ }
+
+ query_tree_walker(query, inline_cte_walker, ctxp,
+ QTW_IGNORE_RANGE_TABLE | QTW_IGNORE_CTE_SUBQUERIES);
+
+ ctx->levelsup--;
+
+ return false;
+ }
+
+ return expression_tree_walker(node, inline_cte_walker, ctxp);
+}
+
+static void inline_cte(PlannerInfo *root, CommonTableExpr *cte)
+{
+ struct inline_cte_walker_ctx ctx;
+ ctx.ctequery = castNode(Query, cte->ctequery);
+ ctx.ctename = cte->ctename;
+ ctx.refcount = cte->cterefcount;
+ ctx.levelsup = -1;
+ ctx.cte = cte;
+ inline_cte_walker((Node *) root->parse, &ctx);
+ /* we must replace all references */
+ Assert(ctx.refcount == 0);
+}
+
+
/*
* SS_process_ctes: process a query's WITH list
*
@@ -1183,6 +1318,23 @@ SS_process_ctes(PlannerInfo *root)
continue;
}
+ /*
+ * Consider inlining the CTE rather than planning it separately.
+ */
+ if (cmdType == CMD_SELECT &&
+ !cte->ctematerialized &&
+ !cte->cterecursive &&
+ !((cte->cterefcount > 1) && contain_volatile_functions(cte->ctequery)) &&
+ (castNode(Query, cte->ctequery)->rowMarks == NIL)
+ )
+ {
+ inline_cte(root, cte);
+
+ /* Make a dummy entry in cte_plan_ids */
+ root->cte_plan_ids = lappend_int(root->cte_plan_ids, -1);
+ continue;
+ }
+
/*
* Copy the source Query node. Probably not necessary, but let's keep
* this similar to make_subplan.
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 7855cff30d..047d3de71c 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -1397,6 +1397,7 @@ typedef struct CommonTableExpr
int location; /* token location, or -1 if unknown */
/* These fields are set during parse analysis: */
bool cterecursive; /* is this CTE actually recursive? */
+ bool ctematerialized; /* is this an optimization fence? */
int cterefcount; /* number of RTEs referencing this CTE
* (excluding internal self-references) */
List *ctecolnames; /* list of output column names */
diff --git a/src/test/regress/expected/rowsecurity.out b/src/test/regress/expected/rowsecurity.out
index bc16ca4c43..0d8d382830 100644
--- a/src/test/regress/expected/rowsecurity.out
+++ b/src/test/regress/expected/rowsecurity.out
@@ -2181,29 +2181,25 @@ EXPLAIN (COSTS OFF) EXECUTE plancache_test;
PREPARE plancache_test2 AS WITH q AS (SELECT * FROM z1 WHERE f_leak(b)) SELECT * FROM q,z2;
EXPLAIN (COSTS OFF) EXECUTE plancache_test2;
- QUERY PLAN
--------------------------------------------------
+ QUERY PLAN
+-----------------------------------------------------
Nested Loop
- CTE q
- -> Seq Scan on z1
- Filter: (((a % 2) = 0) AND f_leak(b))
- -> CTE Scan on q
+ -> Seq Scan on z2
-> Materialize
- -> Seq Scan on z2
-(7 rows)
+ -> Seq Scan on z1
+ Filter: (((a % 2) = 0) AND f_leak(b))
+(5 rows)
PREPARE plancache_test3 AS WITH q AS (SELECT * FROM z2) SELECT * FROM q,z1 WHERE f_leak(z1.b);
EXPLAIN (COSTS OFF) EXECUTE plancache_test3;
QUERY PLAN
-----------------------------------------------------
Nested Loop
- CTE q
- -> Seq Scan on z2
- -> CTE Scan on q
+ -> Seq Scan on z2
-> Materialize
-> Seq Scan on z1
Filter: (((a % 2) = 0) AND f_leak(b))
-(7 rows)
+(5 rows)
SET ROLE regress_rls_group1;
SELECT * FROM z1 WHERE f_leak(b);
@@ -2230,28 +2226,24 @@ EXPLAIN (COSTS OFF) EXECUTE plancache_test;
(2 rows)
EXPLAIN (COSTS OFF) EXECUTE plancache_test2;
- QUERY PLAN
--------------------------------------------------
+ QUERY PLAN
+-----------------------------------------------------
Nested Loop
- CTE q
- -> Seq Scan on z1
- Filter: (((a % 2) = 0) AND f_leak(b))
- -> CTE Scan on q
+ -> Seq Scan on z2
-> Materialize
- -> Seq Scan on z2
-(7 rows)
+ -> Seq Scan on z1
+ Filter: (((a % 2) = 0) AND f_leak(b))
+(5 rows)
EXPLAIN (COSTS OFF) EXECUTE plancache_test3;
QUERY PLAN
-----------------------------------------------------
Nested Loop
- CTE q
- -> Seq Scan on z2
- -> CTE Scan on q
+ -> Seq Scan on z2
-> Materialize
-> Seq Scan on z1
Filter: (((a % 2) = 0) AND f_leak(b))
-(7 rows)
+(5 rows)
SET SESSION AUTHORIZATION regress_rls_carol;
SELECT * FROM z1 WHERE f_leak(b);
@@ -2278,28 +2270,24 @@ EXPLAIN (COSTS OFF) EXECUTE plancache_test;
(2 rows)
EXPLAIN (COSTS OFF) EXECUTE plancache_test2;
- QUERY PLAN
--------------------------------------------------
+ QUERY PLAN
+-----------------------------------------------------
Nested Loop
- CTE q
- -> Seq Scan on z1
- Filter: (((a % 2) = 1) AND f_leak(b))
- -> CTE Scan on q
+ -> Seq Scan on z2
-> Materialize
- -> Seq Scan on z2
-(7 rows)
+ -> Seq Scan on z1
+ Filter: (((a % 2) = 1) AND f_leak(b))
+(5 rows)
EXPLAIN (COSTS OFF) EXECUTE plancache_test3;
QUERY PLAN
-----------------------------------------------------
Nested Loop
- CTE q
- -> Seq Scan on z2
- -> CTE Scan on q
+ -> Seq Scan on z2
-> Materialize
-> Seq Scan on z1
Filter: (((a % 2) = 1) AND f_leak(b))
-(7 rows)
+(5 rows)
SET ROLE regress_rls_group2;
SELECT * FROM z1 WHERE f_leak(b);
@@ -2326,28 +2314,24 @@ EXPLAIN (COSTS OFF) EXECUTE plancache_test;
(2 rows)
EXPLAIN (COSTS OFF) EXECUTE plancache_test2;
- QUERY PLAN
--------------------------------------------------
+ QUERY PLAN
+-----------------------------------------------------
Nested Loop
- CTE q
- -> Seq Scan on z1
- Filter: (((a % 2) = 1) AND f_leak(b))
- -> CTE Scan on q
+ -> Seq Scan on z2
-> Materialize
- -> Seq Scan on z2
-(7 rows)
+ -> Seq Scan on z1
+ Filter: (((a % 2) = 1) AND f_leak(b))
+(5 rows)
EXPLAIN (COSTS OFF) EXECUTE plancache_test3;
QUERY PLAN
-----------------------------------------------------
Nested Loop
- CTE q
- -> Seq Scan on z2
- -> CTE Scan on q
+ -> Seq Scan on z2
-> Materialize
-> Seq Scan on z1
Filter: (((a % 2) = 1) AND f_leak(b))
-(7 rows)
+(5 rows)
--
-- Views should follow policy for view owner.
@@ -2854,13 +2838,11 @@ NOTICE: f_leak => 98f13708210194c475687be6106a3b84
(11 rows)
EXPLAIN (COSTS OFF) WITH cte1 AS (SELECT * FROM t1 WHERE f_leak(b)) SELECT * FROM cte1;
- QUERY PLAN
--------------------------------------------------
- CTE Scan on cte1
- CTE cte1
- -> Seq Scan on t1
- Filter: (((a % 2) = 0) AND f_leak(b))
-(4 rows)
+ QUERY PLAN
+-----------------------------------------
+ Seq Scan on t1
+ Filter: (((a % 2) = 0) AND f_leak(b))
+(2 rows)
WITH cte1 AS (UPDATE t1 SET a = a + 1 RETURNING *) SELECT * FROM cte1; --fail
ERROR: new row violates row-level security policy for table "t1"
diff --git a/src/test/regress/expected/rowtypes.out b/src/test/regress/expected/rowtypes.out
index 30053d07df..c0df2f6333 100644
--- a/src/test/regress/expected/rowtypes.out
+++ b/src/test/regress/expected/rowtypes.out
@@ -1041,14 +1041,11 @@ with r(a,b) as
(values (1,row(1,2)), (1,row(null,null)), (1,null),
(null,row(1,2)), (null,row(null,null)), (null,null) )
select r, r is null as isnull, r is not null as isnotnull from r;
- QUERY PLAN
-----------------------------------------------------------
- CTE Scan on r
- Output: r.*, (r.* IS NULL), (r.* IS NOT NULL)
- CTE r
- -> Values Scan on "*VALUES*"
- Output: "*VALUES*".column1, "*VALUES*".column2
-(5 rows)
+ QUERY PLAN
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ Values Scan on "*VALUES*"
+ Output: ROW("*VALUES*".column1, "*VALUES*".column2), (("*VALUES*".column1 IS NULL) AND ("*VALUES*".column2 IS NOT DISTINCT FROM NULL)), (("*VALUES*".column1 IS NOT NULL) AND ("*VALUES*".column2 IS DISTINCT FROM NULL))
+(2 rows)
with r(a,b) as
(values (1,row(1,2)), (1,row(null,null)), (1,null),
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 744d501e31..882469b79f 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -3135,10 +3135,8 @@ RETURNING *;
Conflict Resolution: UPDATE
Conflict Arbiter Indexes: hat_data_unique_idx
Conflict Filter: ((excluded.hat_color <> 'forbidden'::bpchar) AND (hat_data.* <> excluded.*))
- CTE data
- -> Values Scan on "*VALUES*"
- -> CTE Scan on data
-(7 rows)
+ -> Values Scan on "*VALUES*"
+(5 rows)
SELECT * FROM hat_data WHERE hat_name IN ('h8', 'h9', 'h7') ORDER BY hat_name;
hat_name | hat_color
--
2.17.1
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
@ 2018-08-08 04:55 ` Thomas Munro <thomas.munro@enterprisedb.com>
2018-08-08 05:02 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 112+ messages in thread
From: Thomas Munro @ 2018-08-08 04:55 UTC (permalink / raw)
To: David Fetter <david@fetter.org>; +Cc: Andrew Gierth <andrew@tao11.riddles.org.uk>; pgsql-hackers
On Fri, Jul 27, 2018 at 8:10 PM, David Fetter <david@fetter.org> wrote:
> On Fri, Jul 27, 2018 at 02:55:26PM +1200, Thomas Munro wrote:
>> On Thu, Jul 26, 2018 at 7:14 AM, David Fetter <david@fetter.org> wrote:
>> > Please find attached the next version, which passes 'make check'.
>>
>> ... but not 'make check-world' (contrib/postgres_fdw's EXPLAIN is different).
>
> Please find attached a patch that does.
>
> It doesn't always pass make installcheck-world, but I need to sleep
> rather than investigate that at the moment.
One observation I wanted to share: CTE scans inhibit parallelism today
(something we might eventually want to fix with shared tuplestores).
This patch therefore allows parallelism in some WITH queries, which
seems like a very valuable thing. Example:
postgres=# create table foo as select generate_series(1, 1000000) i;
SELECT 1000000
postgres=# create table bar as select generate_series(1, 1000000) i;
SELECT 1000000
postgres=# create table baz as select generate_series(1, 1000000) i;
SELECT 1000000
postgres=# analyze;
ANALYZE
=== unpatched master ===
postgres=# explain analyze with cte as (select * from foo join bar
using (i)) select count(*) from cte join baz using (i);
QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------------
Aggregate (cost=149531.00..149531.01 rows=1 width=8) (actual
time=4400.951..4400.951 rows=1 loops=1)
CTE cte
-> Hash Join (cost=30832.00..70728.00 rows=1000000 width=4)
(actual time=551.243..1961.319 rows=1000000 loops=1)
Hash Cond: (foo.i = bar.i)
-> Seq Scan on foo (cost=0.00..14425.00 rows=1000000
width=4) (actual time=0.048..219.238 rows=1000000 loops=1)
-> Hash (cost=14425.00..14425.00 rows=1000000 width=4)
(actual time=550.477..550.478 rows=1000000 loops=1)
Buckets: 131072 Batches: 16 Memory Usage: 3227kB
-> Seq Scan on bar (cost=0.00..14425.00
rows=1000000 width=4) (actual time=0.031..213.238 rows=1000000
loops=1)
-> Hash Join (cost=30832.00..76303.00 rows=1000000 width=0)
(actual time=1090.162..4279.945 rows=1000000 loops=1)
Hash Cond: (cte.i = baz.i)
-> CTE Scan on cte (cost=0.00..20000.00 rows=1000000
width=4) (actual time=551.247..2564.529 rows=1000000 loops=1)
-> Hash (cost=14425.00..14425.00 rows=1000000 width=4)
(actual time=538.833..538.833 rows=1000000 loops=1)
Buckets: 131072 Batches: 16 Memory Usage: 3227kB
-> Seq Scan on baz (cost=0.00..14425.00 rows=1000000
width=4) (actual time=0.039..208.658 rows=1000000 loops=1)
Planning Time: 0.291 ms
Execution Time: 4416.732 ms
(16 rows)
=== 0001-Inlining-CTEs-v0005.patch ===
postgres=# explain analyze with cte as (select * from foo join bar
using (i)) select count(*) from cte join baz using (i);
QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------------------
Finalize Aggregate (cost=57854.78..57854.79 rows=1 width=8) (actual
time=1441.663..1441.664 rows=1 loops=1)
-> Gather (cost=57854.57..57854.78 rows=2 width=8) (actual
time=1440.506..1474.974 rows=3 loops=1)
Workers Planned: 2
Workers Launched: 2
-> Partial Aggregate (cost=56854.57..56854.58 rows=1
width=8) (actual time=1435.017..1435.018 rows=1 loops=3)
-> Parallel Hash Join (cost=30856.01..55812.90
rows=416667 width=0) (actual time=1135.164..1393.437 rows=333333
loops=3)
Hash Cond: (foo.i = baz.i)
-> Parallel Hash Join (cost=15428.00..32202.28
rows=416667 width=8) (actual time=457.786..753.374 rows=333333
loops=3)
Hash Cond: (foo.i = bar.i)
-> Parallel Seq Scan on foo
(cost=0.00..8591.67 rows=416667 width=4) (actual time=0.094..87.666
rows=333333 loops=3)
-> Parallel Hash (cost=8591.67..8591.67
rows=416667 width=4) (actual time=217.222..217.222 rows=333333
loops=3)
Buckets: 131072 Batches: 16 Memory
Usage: 3520kB
-> Parallel Seq Scan on bar
(cost=0.00..8591.67 rows=416667 width=4) (actual time=0.061..84.631
rows=333333 loops=3)
-> Parallel Hash (cost=8591.67..8591.67
rows=416667 width=4) (actual time=227.240..227.241 rows=333333
loops=3)
Buckets: 131072 Batches: 16 Memory Usage: 3520kB
-> Parallel Seq Scan on baz
(cost=0.00..8591.67 rows=416667 width=4) (actual time=0.060..84.270
rows=333333 loops=3)
Planning Time: 0.407 ms
Execution Time: 1475.113 ms
(18 rows)
--
Thomas Munro
http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-08-08 04:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
@ 2018-08-08 05:02 ` Andres Freund <andres@anarazel.de>
2019-03-13 00:45 ` Re: Early WIP/PoC for inlining CTEs Tatsuo Ishii <ishii@sraoss.co.jp>
0 siblings, 1 reply; 112+ messages in thread
From: Andres Freund @ 2018-08-08 05:02 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@enterprisedb.com>; +Cc: David Fetter <david@fetter.org>; Andrew Gierth <andrew@tao11.riddles.org.uk>; pgsql-hackers
Hi,
On 2018-08-08 16:55:22 +1200, Thomas Munro wrote:
> On Fri, Jul 27, 2018 at 8:10 PM, David Fetter <david@fetter.org> wrote:
> > On Fri, Jul 27, 2018 at 02:55:26PM +1200, Thomas Munro wrote:
> >> On Thu, Jul 26, 2018 at 7:14 AM, David Fetter <david@fetter.org> wrote:
> >> > Please find attached the next version, which passes 'make check'.
> >>
> >> ... but not 'make check-world' (contrib/postgres_fdw's EXPLAIN is different).
> >
> > Please find attached a patch that does.
> >
> > It doesn't always pass make installcheck-world, but I need to sleep
> > rather than investigate that at the moment.
>
> One observation I wanted to share: CTE scans inhibit parallelism today
> (something we might eventually want to fix with shared tuplestores).
> This patch therefore allows parallelism in some WITH queries, which
> seems like a very valuable thing.
Might be interesting to see how big a difference it makes for
TPC-DS. Currently the results are bad (as in many queries don't finish
in a relevant time) because it uses CTEs so widely, and there's often
predicates outside the CTE that could be pushed down.
- Andres
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-08-08 04:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-08-08 05:02 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
@ 2019-03-13 00:45 ` Tatsuo Ishii <ishii@sraoss.co.jp>
0 siblings, 0 replies; 112+ messages in thread
From: Tatsuo Ishii @ 2019-03-13 00:45 UTC (permalink / raw)
To: andres@anarazel.de; +Cc: thomas.munro@enterprisedb.com; david@fetter.org; andrew@tao11.riddles.org.uk; pgsql-hackers
> On 2018-08-08 16:55:22 +1200, Thomas Munro wrote:
>> On Fri, Jul 27, 2018 at 8:10 PM, David Fetter <david@fetter.org> wrote:
>> > On Fri, Jul 27, 2018 at 02:55:26PM +1200, Thomas Munro wrote:
>> >> On Thu, Jul 26, 2018 at 7:14 AM, David Fetter <david@fetter.org> wrote:
>> >> > Please find attached the next version, which passes 'make check'.
>> >>
>> >> ... but not 'make check-world' (contrib/postgres_fdw's EXPLAIN is different).
>> >
>> > Please find attached a patch that does.
>> >
>> > It doesn't always pass make installcheck-world, but I need to sleep
>> > rather than investigate that at the moment.
>>
>> One observation I wanted to share: CTE scans inhibit parallelism today
>> (something we might eventually want to fix with shared tuplestores).
>> This patch therefore allows parallelism in some WITH queries, which
>> seems like a very valuable thing.
>
> Might be interesting to see how big a difference it makes for
> TPC-DS. Currently the results are bad (as in many queries don't finish
> in a relevant time) because it uses CTEs so widely, and there's often
> predicates outside the CTE that could be pushed down.
Now that the patch was committed, I played with TPCS-DS and found at
least one of their queries gets speedup. Query 2 runs 2 times faster
than 11. In 12, it seems that CTE wscs is pushed down.
with wscs as
(select sold_date_sk
,sales_price
from (select ws_sold_date_sk sold_date_sk
,ws_ext_sales_price sales_price
from web_sales
union all
select cs_sold_date_sk sold_date_sk
,cs_ext_sales_price sales_price
from catalog_sales) as s1),
wswscs as
(select d_week_seq,
sum(case when (d_day_name='Sunday') then sales_price else null end) sun_sales,
sum(case when (d_day_name='Monday') then sales_price else null end) mon_sales,
sum(case when (d_day_name='Tuesday') then sales_price else null end) tue_sales,
sum(case when (d_day_name='Wednesday') then sales_price else null end) wed_sales,
sum(case when (d_day_name='Thursday') then sales_price else null end) thu_sales,
sum(case when (d_day_name='Friday') then sales_price else null end) fri_sales,
sum(case when (d_day_name='Saturday') then sales_price else null end) sat_sales
from wscs
,date_dim
where d_date_sk = sold_date_sk
group by d_week_seq)
select d_week_seq1
,round(sun_sales1/sun_sales2,2)
,round(mon_sales1/mon_sales2,2)
,round(tue_sales1/tue_sales2,2)
,round(wed_sales1/wed_sales2,2)
,round(thu_sales1/thu_sales2,2)
,round(fri_sales1/fri_sales2,2)
,round(sat_sales1/sat_sales2,2)
from
(select wswscs.d_week_seq d_week_seq1
,sun_sales sun_sales1
,mon_sales mon_sales1
,tue_sales tue_sales1
,wed_sales wed_sales1
,thu_sales thu_sales1
,fri_sales fri_sales1
,sat_sales sat_sales1
from wswscs,date_dim
where date_dim.d_week_seq = wswscs.d_week_seq and
d_year = 1998) y,
(select wswscs.d_week_seq d_week_seq2
,sun_sales sun_sales2
,mon_sales mon_sales2
,tue_sales tue_sales2
,wed_sales wed_sales2
,thu_sales thu_sales2
,fri_sales fri_sales2
,sat_sales sat_sales2
from wswscs
,date_dim
where date_dim.d_week_seq = wswscs.d_week_seq and
d_year = 1998+1) z
where d_week_seq1=d_week_seq2-53
order by d_week_seq1;
Here's the 12's plan:
QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Sort (cost=118929.39..118929.43 rows=13 width=228) (actual time=792.588..792.710 rows=2513 loops=1)
Sort Key: wswscs.d_week_seq
Sort Method: quicksort Memory: 323kB
CTE wswscs
-> Finalize GroupAggregate (cost=110164.09..113672.71 rows=10447 width=228) (actual time=766.232..768.415 rows=263 loops=1)
Group Key: date_dim_2.d_week_seq
-> Gather Merge (cost=110164.09..112601.89 rows=20894 width=228) (actual time=766.209..767.158 rows=789 loops=1)
Workers Planned: 2
Workers Launched: 2
-> Sort (cost=109164.06..109190.18 rows=10447 width=228) (actual time=763.059..763.078 rows=263 loops=3)
Sort Key: date_dim_2.d_week_seq
Sort Method: quicksort Memory: 160kB
Worker 0: Sort Method: quicksort Memory: 160kB
Worker 1: Sort Method: quicksort Memory: 160kB
-> Partial HashAggregate (cost=108179.39..108466.69 rows=10447 width=228) (actual time=762.202..762.889 rows=263 loops=3)
Group Key: date_dim_2.d_week_seq
-> Parallel Hash Join (cost=2371.82..74413.79 rows=900416 width=20) (actual time=17.166..424.834 rows=717854 loops=3)
Hash Cond: (catalog_sales.cs_sold_date_sk = date_dim_2.d_date_sk)
-> Parallel Append (cost=0.00..69678.24 rows=900416 width=10) (actual time=0.029..248.992 rows=720311 loops=3)
-> Parallel Seq Scan on catalog_sales (cost=0.00..43411.73 rows=600673 width=10) (actual time=0.018..130.163 rows=480516 loops=3)
-> Parallel Seq Scan on web_sales (cost=0.00..21764.43 rows=299743 width=10) (actual time=0.026..95.629 rows=359692 loops=2)
-> Parallel Hash (cost=1834.70..1834.70 rows=42970 width=18) (actual time=16.610..16.610 rows=24350 loops=3)
Buckets: 131072 Batches: 1 Memory Usage: 5056kB
-> Parallel Seq Scan on date_dim date_dim_2 (cost=0.00..1834.70 rows=42970 width=18) (actual time=0.020..7.617 rows=24350 loops=3)
-> Hash Join (cost=5007.74..5256.44 rows=13 width=228) (actual time=785.300..792.123 rows=2513 loops=1)
Hash Cond: (wswscs.d_week_seq = date_dim.d_week_seq)
-> CTE Scan on wswscs (cost=0.00..208.94 rows=10447 width=228) (actual time=766.236..766.263 rows=263 loops=1)
-> Hash (cost=5007.58..5007.58 rows=13 width=232) (actual time=19.033..19.033 rows=2513 loops=1)
Buckets: 4096 (originally 1024) Batches: 1 (originally 1) Memory Usage: 288kB
-> Merge Join (cost=5001.97..5007.58 rows=13 width=232) (actual time=17.739..18.210 rows=2513 loops=1)
Merge Cond: (((wswscs_1.d_week_seq - 53)) = date_dim.d_week_seq)
-> Sort (cost=2668.33..2669.24 rows=365 width=228) (actual time=9.906..9.924 rows=365 loops=1)
Sort Key: ((wswscs_1.d_week_seq - 53))
Sort Method: quicksort Memory: 76kB
-> Hash Join (cost=2322.68..2652.79 rows=365 width=228) (actual time=7.864..9.764 rows=365 loops=1)
Hash Cond: (wswscs_1.d_week_seq = date_dim_1.d_week_seq)
-> CTE Scan on wswscs wswscs_1 (cost=0.00..208.94 rows=10447 width=228) (actual time=0.001..2.287 rows=263 loops=1)
-> Hash (cost=2318.11..2318.11 rows=365 width=4) (actual time=7.389..7.389 rows=365 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 21kB
-> Seq Scan on date_dim date_dim_1 (cost=0.00..2318.11 rows=365 width=4) (actual time=3.876..7.348 rows=365 loops=1)
Filter: (d_year = 1999)
Rows Removed by Filter: 72684
-> Sort (cost=2333.65..2334.56 rows=365 width=4) (actual time=7.824..7.930 rows=2514 loops=1)
Sort Key: date_dim.d_week_seq
Sort Method: quicksort Memory: 42kB
-> Seq Scan on date_dim (cost=0.00..2318.11 rows=365 width=4) (actual time=3.950..7.765 rows=365 loops=1)
Filter: (d_year = 1998)
Rows Removed by Filter: 72684
Planning Time: 0.956 ms
Execution Time: 794.072 ms
(50 rows)
Here's 11's plan:
QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------------------------
Sort (cost=227105.32..227105.35 rows=13 width=228) (actual time=2019.071..2019.163 rows=2513 loops=1)
Sort Key: wswscs.d_week_seq
Sort Method: quicksort Memory: 323kB
CTE wscs
-> Append (cost=0.00..88586.64 rows=2160976 width=10) (actual time=0.015..521.937 rows=2160932 loops=1)
-> Seq Scan on web_sales (cost=0.00..25960.84 rows=719384 width=10) (actual time=0.014..137.557 rows=719384 loops=1)
-> Seq Scan on catalog_sales (cost=0.00..51820.92 rows=1441592 width=10) (actual time=0.011..269.559 rows=1441548 loops=1)
CTE wswscs
-> HashAggregate (cost=132977.62..133264.03 rows=10415 width=228) (actual time=1996.856..1997.387 rows=263 loops=1)
Group Key: date_dim_2.d_week_seq
-> Hash Join (cost=3048.60..51941.02 rows=2160976 width=28) (actual time=36.414..1323.387 rows=2153563 loops=1)
Hash Cond: (wscs.sold_date_sk = date_dim_2.d_date_sk)
-> CTE Scan on wscs (cost=0.00..43219.52 rows=2160976 width=18) (actual time=0.017..942.090 rows=2160932 loops=1)
-> Hash (cost=2135.49..2135.49 rows=73049 width=18) (actual time=35.870..35.871 rows=73049 loops=1)
Buckets: 131072 Batches: 1 Memory Usage: 4734kB
-> Seq Scan on date_dim date_dim_2 (cost=0.00..2135.49 rows=73049 width=18) (actual time=0.011..17.749 rows=73049 loops=1)
-> Hash Join (cost=5006.47..5254.41 rows=13 width=228) (actual time=2012.260..2018.602 rows=2513 loops=1)
Hash Cond: (wswscs.d_week_seq = date_dim.d_week_seq)
-> CTE Scan on wswscs (cost=0.00..208.30 rows=10415 width=228) (actual time=1996.858..1996.876 rows=263 loops=1)
-> Hash (cost=5006.31..5006.31 rows=13 width=232) (actual time=15.380..15.380 rows=2513 loops=1)
Buckets: 4096 (originally 1024) Batches: 1 (originally 1) Memory Usage: 288kB
-> Merge Join (cost=5000.73..5006.31 rows=13 width=232) (actual time=14.272..14.683 rows=2513 loops=1)
Merge Cond: (((wswscs_1.d_week_seq - 53)) = date_dim.d_week_seq)
-> Sort (cost=2667.18..2668.09 rows=363 width=228) (actual time=7.374..7.393 rows=365 loops=1)
Sort Key: ((wswscs_1.d_week_seq - 53))
Sort Method: quicksort Memory: 76kB
-> Hash Join (cost=2322.65..2651.75 rows=363 width=228) (actual time=6.386..7.154 rows=365 loops=1)
Hash Cond: (wswscs_1.d_week_seq = date_dim_1.d_week_seq)
-> CTE Scan on wswscs wswscs_1 (cost=0.00..208.30 rows=10415 width=228) (actual time=0.000..0.707 rows=263 loops=1)
-> Hash (cost=2318.11..2318.11 rows=363 width=4) (actual time=6.367..6.367 rows=365 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 21kB
-> Seq Scan on date_dim date_dim_1 (cost=0.00..2318.11 rows=363 width=4) (actual time=3.000..6.330 rows=365 loops=1)
Filter: (d_year = 1999)
Rows Removed by Filter: 72684
-> Sort (cost=2333.55..2334.45 rows=363 width=4) (actual time=6.890..6.975 rows=2514 loops=1)
Sort Key: date_dim.d_week_seq
Sort Method: quicksort Memory: 42kB
-> Seq Scan on date_dim (cost=0.00..2318.11 rows=363 width=4) (actual time=3.832..6.841 rows=365 loops=1)
Filter: (d_year = 1998)
Rows Removed by Filter: 72684
Planning Time: 0.962 ms
Execution Time: 2027.758 ms
(42 rows)
BTW, in my small TPC-DS environment (2GB), only two queries were not
finished within 30 minutes (query 4 and 11). My guess is these seem to
be suffered from statistic errors (I got this hint from Kaigai-san's
slide: https://www.slideshare.net/kaigai/tpcdspostgresql, especially
page 20. Unfortunately the slide is in Japanese). And it leads to
choosing a bad plan: nested loop join. If I disable nested loop join,
these two queries finished quickly.
Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
@ 2018-10-01 12:55 ` Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
1 sibling, 1 reply; 112+ messages in thread
From: Andreas Karlsson @ 2018-10-01 12:55 UTC (permalink / raw)
To: David Fetter <david@fetter.org>; Thomas Munro <thomas.munro@enterprisedb.com>; +Cc: Andrew Gierth <andrew@tao11.riddles.org.uk>; pgsql-hackers
On 07/27/2018 10:10 AM, David Fetter wrote:
> On Fri, Jul 27, 2018 at 02:55:26PM +1200, Thomas Munro wrote:
>> On Thu, Jul 26, 2018 at 7:14 AM, David Fetter <david@fetter.org> wrote:
>>> Please find attached the next version, which passes 'make check'.
>>
>> ... but not 'make check-world' (contrib/postgres_fdw's EXPLAIN is different).
>
> Please find attached a patch that does.
>
> It doesn't always pass make installcheck-world, but I need to sleep
> rather than investigate that at the moment.
I took a quick look at this patch and I have a couple of comments.
1) Is it really safe, for backwards compatibility reasons, to inline
when there are volatile functions? I imagine that it is possible that
there are people who rely on that volatile functions inside a CTE are
always run.
Imagine this case:
WITH cte AS (
SELECT x, volatile_f(x) FROM tab ORDER BY x
)
SELECT * FROM cte LIMIT 10;
It could change behavior if volatile_f() has side effects and we inline
the CTE. Is backwards compatibility for cases like this worth
preserving? People can get the old behavior by adding OFFSET 0 or
MATERIALIZED, but existing code would break.
2) The code in inline_cte_walker() is quite finicky. It looks correct to
me but it is hard to follow and some simple bug could easily be hiding
in there. I wonder if this code could be rewritten in some way to make
it easier to follow.
3) Can you recall what the failing test was because I have so far not
managed to reproduce it?
Andreas
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
@ 2018-10-02 10:03 ` Andreas Karlsson <andreas@proxel.se>
2018-10-03 15:57 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 2 replies; 112+ messages in thread
From: Andreas Karlsson @ 2018-10-02 10:03 UTC (permalink / raw)
To: David Fetter <david@fetter.org>; Thomas Munro <thomas.munro@enterprisedb.com>; +Cc: Andrew Gierth <andrew@tao11.riddles.org.uk>; pgsql-hackers
Hi,
Here is an updated patch which adds some simple syntax for adding the
optimization barrier. For example:
WITH x AS MATERIALIZED (
SELECT 1
)
SELECT * FROM x;
Andreas
Attachments:
[text/x-patch] inlining-ctes-v5.patch (20.6K, ../../e02af96a-3e25-5de4-09a4-a7ebf3452184@proxel.se/2-inlining-ctes-v5.patch)
download | inline diff:
diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out
index 21a2ef5ad3a..017b73e0a29 100644
--- a/contrib/postgres_fdw/expected/postgres_fdw.out
+++ b/contrib/postgres_fdw/expected/postgres_fdw.out
@@ -1888,7 +1888,7 @@ SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) ORDER BY t1.c3, t
-- join in CTE
EXPLAIN (VERBOSE, COSTS OFF)
-WITH t (c1_1, c1_3, c2_1) AS (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
+WITH t (c1_1, c1_3, c2_1) AS MATERIALIZED (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
QUERY PLAN
-------------------------------------------------------------------------------------------------------------------------------------
Limit
@@ -1905,7 +1905,7 @@ WITH t (c1_1, c1_3, c2_1) AS (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2
Output: t.c1_1, t.c2_1, t.c1_3
(12 rows)
-WITH t (c1_1, c1_3, c2_1) AS (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
+WITH t (c1_1, c1_3, c2_1) AS MATERIALIZED (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
c1_1 | c2_1
------+------
101 | 101
diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql
index 88c4cb4783f..3032aed34f9 100644
--- a/contrib/postgres_fdw/sql/postgres_fdw.sql
+++ b/contrib/postgres_fdw/sql/postgres_fdw.sql
@@ -501,8 +501,8 @@ SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) ORDER BY t1.c3, t
SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) ORDER BY t1.c3, t1.c1 OFFSET 100 LIMIT 10 FOR SHARE;
-- join in CTE
EXPLAIN (VERBOSE, COSTS OFF)
-WITH t (c1_1, c1_3, c2_1) AS (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
-WITH t (c1_1, c1_3, c2_1) AS (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
+WITH t (c1_1, c1_3, c2_1) AS MATERIALIZED (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
+WITH t (c1_1, c1_3, c2_1) AS MATERIALIZED (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
-- ctid with whole-row reference
EXPLAIN (VERBOSE, COSTS OFF)
SELECT t1.ctid, t1, t2, t1.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) ORDER BY t1.c3, t1.c1 OFFSET 100 LIMIT 10;
diff --git a/doc/src/sgml/ref/select.sgml b/doc/src/sgml/ref/select.sgml
index 4db8142afaa..43f1b0e3f15 100644
--- a/doc/src/sgml/ref/select.sgml
+++ b/doc/src/sgml/ref/select.sgml
@@ -72,7 +72,7 @@ SELECT [ ALL | DISTINCT [ ON ( <replaceable class="parameter">expression</replac
<phrase>and <replaceable class="parameter">with_query</replaceable> is:</phrase>
- <replaceable class="parameter">with_query_name</replaceable> [ ( <replaceable class="parameter">column_name</replaceable> [, ...] ) ] AS ( <replaceable class="parameter">select</replaceable> | <replaceable class="parameter">values</replaceable> | <replaceable class="parameter">insert</replaceable> | <replaceable class="parameter">update</replaceable> | <replaceable class="parameter">delete</replaceable> )
+ <replaceable class="parameter">with_query_name</replaceable> [ ( <replaceable class="parameter">column_name</replaceable> [, ...] ) ] AS [ MATERIALIZED ] ( <replaceable class="parameter">select</replaceable> | <replaceable class="parameter">values</replaceable> | <replaceable class="parameter">insert</replaceable> | <replaceable class="parameter">update</replaceable> | <replaceable class="parameter">delete</replaceable> )
TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ]
</synopsis>
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 925cb8f3800..84435acb83e 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2542,6 +2542,7 @@ _copyCommonTableExpr(const CommonTableExpr *from)
COPY_NODE_FIELD(ctequery);
COPY_LOCATION_FIELD(location);
COPY_SCALAR_FIELD(cterecursive);
+ COPY_SCALAR_FIELD(ctematerialized);
COPY_SCALAR_FIELD(cterefcount);
COPY_NODE_FIELD(ctecolnames);
COPY_NODE_FIELD(ctecoltypes);
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index 3bb91c95958..65875d90201 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2794,6 +2794,7 @@ _equalCommonTableExpr(const CommonTableExpr *a, const CommonTableExpr *b)
COMPARE_NODE_FIELD(ctequery);
COMPARE_LOCATION_FIELD(location);
COMPARE_SCALAR_FIELD(cterecursive);
+ COMPARE_SCALAR_FIELD(ctematerialized);
COMPARE_SCALAR_FIELD(cterefcount);
COMPARE_NODE_FIELD(ctecolnames);
COMPARE_NODE_FIELD(ctecoltypes);
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 22dbae15d3b..56bfdcc5d10 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -3094,6 +3094,7 @@ _outCommonTableExpr(StringInfo str, const CommonTableExpr *node)
WRITE_NODE_FIELD(ctequery);
WRITE_LOCATION_FIELD(location);
WRITE_BOOL_FIELD(cterecursive);
+ WRITE_BOOL_FIELD(ctematerialized);
WRITE_INT_FIELD(cterefcount);
WRITE_NODE_FIELD(ctecolnames);
WRITE_NODE_FIELD(ctecoltypes);
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index ce556580a5f..4b47d2e7d2f 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -420,6 +420,7 @@ _readCommonTableExpr(void)
READ_NODE_FIELD(ctequery);
READ_LOCATION_FIELD(location);
READ_BOOL_FIELD(cterecursive);
+ READ_BOOL_FIELD(ctematerialized);
READ_INT_FIELD(cterefcount);
READ_NODE_FIELD(ctecolnames);
READ_NODE_FIELD(ctecoltypes);
diff --git a/src/backend/optimizer/plan/subselect.c b/src/backend/optimizer/plan/subselect.c
index 83008d76619..5aa242ff26f 100644
--- a/src/backend/optimizer/plan/subselect.c
+++ b/src/backend/optimizer/plan/subselect.c
@@ -1146,6 +1146,141 @@ hash_ok_operator(OpExpr *expr)
}
+struct inline_cte_walker_ctx
+{
+ const char *ctename;
+ int levelsup;
+ int refcount;
+ Query *ctequery;
+ CommonTableExpr *cte;
+};
+
+static bool inline_cte_walker(Node *node, void *ctxp)
+{
+ struct inline_cte_walker_ctx *ctx = ctxp;
+
+ if (!node)
+ return false;
+
+ if (IsA(node, Query))
+ {
+ /*
+ * This one is a bit tricky. It's our job to handle the recursion here,
+ * but we do some of the lifting normally handled by query_tree_walker
+ * in order to get the sequence of operations right.
+ *
+ * First, if the Query we're looking at is the one containing our CTE
+ * definition, then we don't need to recurse into our own CTE or CTEs
+ * that are earlier in the list than ours (since cteList has been
+ * sorted for us into dependency order). We could check whether a
+ * nested query is hiding ours, but that seems too much of an edge case
+ * to be worth optimizing (the levelsup check will ensure we don't
+ * replace any CTEs improperly). So we scan the cteList ourselves
+ * rather than having query_tree_walker do it.
+ *
+ * Second, we want to walk the rangetable _before_ replacing any
+ * RTE_CTE nodes, in order to avoid re-walking the subquery we just
+ * inlined. (range_table_walker, if told to visit the RTE nodes at all,
+ * visits them before their content.) So we have range_table_walker
+ * ignore the RTE nodes themselves and only walk their contents.
+ *
+ * Third, we scan the rangetable for RTE_CTE nodes to replace.
+ *
+ * Fourth, we use query_tree_walker to find and walk the rest of the
+ * query, telling it to ignore the rangetable and CTEs.
+ *
+ * Note that ctx->levelsup is -1 on entry the first time, since we need
+ * the incremented value to be 0 when scanning the content of the query
+ * containing the definition.
+ */
+ Query *query = castNode(Query, node);
+ ListCell *lc;
+ bool do_replace = ctx->levelsup >= 0;
+
+ ctx->levelsup++;
+
+ foreach (lc, query->cteList)
+ {
+ CommonTableExpr *cte = lfirst_node(CommonTableExpr, lc);
+
+ if (!do_replace && strcmp(cte->ctename, ctx->ctename) == 0)
+ do_replace = true;
+ else if (do_replace)
+ inline_cte_walker(cte->ctequery, ctxp);
+ }
+
+ range_table_walker(query->rtable, inline_cte_walker, ctxp, 0);
+
+ foreach (lc, query->rtable)
+ {
+ RangeTblEntry *rte = lfirst_node(RangeTblEntry, lc);
+
+ if (rte->rtekind == RTE_CTE &&
+ strcmp(rte->ctename, ctx->ctename) == 0 &&
+ rte->ctelevelsup == ctx->levelsup)
+ {
+ Query *newquery = ctx->ctequery;
+
+ /*
+ * We need to do some work here that view rewrite does not, and
+ * in turn we do not do some work that view rewrite does.
+ *
+ * Firstly, views can't have outer references but CTEs can
+ * (especially in the case of CTEs referencing other CTEs), so
+ * we need to fix up all levelsup attributes inside the CTE
+ * query.
+ *
+ * Secondly, views (and explicit subqueries) currently have
+ * different behaviour w.r.t. SELECT FOR UPDATE than CTEs do. A
+ * FOR UPDATE clause is treated as extending into views and
+ * subqueries, but not into CTEs. We preserve this distinction
+ * by not trying to push rowmarks into the new subquery.
+ *
+ * We avoid copyObject if possible because subquery processing
+ * copies the query too.
+ */
+ if (ctx->levelsup > 0)
+ {
+ newquery = copyObject(newquery);
+ IncrementVarSublevelsUp((Node *) newquery, ctx->levelsup, 1);
+ }
+
+ /*
+ * Here's where we do the actual substitution.
+ */
+ rte->rtekind = RTE_SUBQUERY;
+ rte->subquery = newquery;
+ rte->security_barrier = false;
+
+ ctx->refcount--;
+ }
+ }
+
+ query_tree_walker(query, inline_cte_walker, ctxp,
+ QTW_IGNORE_RANGE_TABLE | QTW_IGNORE_CTE_SUBQUERIES);
+
+ ctx->levelsup--;
+
+ return false;
+ }
+
+ return expression_tree_walker(node, inline_cte_walker, ctxp);
+}
+
+static void inline_cte(PlannerInfo *root, CommonTableExpr *cte)
+{
+ struct inline_cte_walker_ctx ctx;
+ ctx.ctequery = castNode(Query, cte->ctequery);
+ ctx.ctename = cte->ctename;
+ ctx.refcount = cte->cterefcount;
+ ctx.levelsup = -1;
+ ctx.cte = cte;
+ inline_cte_walker((Node *) root->parse, &ctx);
+ /* we must replace all references */
+ Assert(ctx.refcount == 0);
+}
+
+
/*
* SS_process_ctes: process a query's WITH list
*
@@ -1183,6 +1318,23 @@ SS_process_ctes(PlannerInfo *root)
continue;
}
+ /*
+ * Consider inlining the CTE rather than planning it separately.
+ */
+ if (cmdType == CMD_SELECT &&
+ !cte->ctematerialized &&
+ !cte->cterecursive &&
+ !((cte->cterefcount > 1) && contain_volatile_functions(cte->ctequery)) &&
+ (castNode(Query, cte->ctequery)->rowMarks == NIL)
+ )
+ {
+ inline_cte(root, cte);
+
+ /* Make a dummy entry in cte_plan_ids */
+ root->cte_plan_ids = lappend_int(root->cte_plan_ids, -1);
+ continue;
+ }
+
/*
* Copy the source Query node. Probably not necessary, but let's keep
* this similar to make_subplan.
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index ce60e99cff5..09b263fdc40 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -518,7 +518,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
Bit ConstBit BitWithLength BitWithoutLength
%type <str> character
%type <str> extract_arg
-%type <boolean> opt_varying opt_timezone opt_no_inherit
+%type <boolean> opt_varying opt_timezone opt_no_inherit opt_materialized
%type <ival> Iconst SignedIconst
%type <str> Sconst comment_text notify_payload
@@ -11383,17 +11383,23 @@ cte_list:
| cte_list ',' common_table_expr { $$ = lappend($1, $3); }
;
-common_table_expr: name opt_name_list AS '(' PreparableStmt ')'
+common_table_expr: name opt_name_list AS opt_materialized '(' PreparableStmt ')'
{
CommonTableExpr *n = makeNode(CommonTableExpr);
n->ctename = $1;
n->aliascolnames = $2;
- n->ctequery = $5;
+ n->ctequery = $6;
+ n->ctematerialized = $4;
n->location = @1;
$$ = (Node *) n;
}
;
+opt_materialized:
+ MATERIALIZED { $$ = true; }
+ | /*EMPTY*/ { $$ = false; }
+ ;
+
opt_with_clause:
with_clause { $$ = $1; }
| /*EMPTY*/ { $$ = NULL; }
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 200df8e6595..03927f7cf70 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -1410,6 +1410,7 @@ typedef struct CommonTableExpr
int location; /* token location, or -1 if unknown */
/* These fields are set during parse analysis: */
bool cterecursive; /* is this CTE actually recursive? */
+ bool ctematerialized; /* is this an optimization fence? */
int cterefcount; /* number of RTEs referencing this CTE
* (excluding internal self-references) */
List *ctecolnames; /* list of output column names */
diff --git a/src/test/regress/expected/rowsecurity.out b/src/test/regress/expected/rowsecurity.out
index bc16ca4c43f..0d8d3828300 100644
--- a/src/test/regress/expected/rowsecurity.out
+++ b/src/test/regress/expected/rowsecurity.out
@@ -2181,29 +2181,25 @@ EXPLAIN (COSTS OFF) EXECUTE plancache_test;
PREPARE plancache_test2 AS WITH q AS (SELECT * FROM z1 WHERE f_leak(b)) SELECT * FROM q,z2;
EXPLAIN (COSTS OFF) EXECUTE plancache_test2;
- QUERY PLAN
--------------------------------------------------
+ QUERY PLAN
+-----------------------------------------------------
Nested Loop
- CTE q
- -> Seq Scan on z1
- Filter: (((a % 2) = 0) AND f_leak(b))
- -> CTE Scan on q
+ -> Seq Scan on z2
-> Materialize
- -> Seq Scan on z2
-(7 rows)
+ -> Seq Scan on z1
+ Filter: (((a % 2) = 0) AND f_leak(b))
+(5 rows)
PREPARE plancache_test3 AS WITH q AS (SELECT * FROM z2) SELECT * FROM q,z1 WHERE f_leak(z1.b);
EXPLAIN (COSTS OFF) EXECUTE plancache_test3;
QUERY PLAN
-----------------------------------------------------
Nested Loop
- CTE q
- -> Seq Scan on z2
- -> CTE Scan on q
+ -> Seq Scan on z2
-> Materialize
-> Seq Scan on z1
Filter: (((a % 2) = 0) AND f_leak(b))
-(7 rows)
+(5 rows)
SET ROLE regress_rls_group1;
SELECT * FROM z1 WHERE f_leak(b);
@@ -2230,28 +2226,24 @@ EXPLAIN (COSTS OFF) EXECUTE plancache_test;
(2 rows)
EXPLAIN (COSTS OFF) EXECUTE plancache_test2;
- QUERY PLAN
--------------------------------------------------
+ QUERY PLAN
+-----------------------------------------------------
Nested Loop
- CTE q
- -> Seq Scan on z1
- Filter: (((a % 2) = 0) AND f_leak(b))
- -> CTE Scan on q
+ -> Seq Scan on z2
-> Materialize
- -> Seq Scan on z2
-(7 rows)
+ -> Seq Scan on z1
+ Filter: (((a % 2) = 0) AND f_leak(b))
+(5 rows)
EXPLAIN (COSTS OFF) EXECUTE plancache_test3;
QUERY PLAN
-----------------------------------------------------
Nested Loop
- CTE q
- -> Seq Scan on z2
- -> CTE Scan on q
+ -> Seq Scan on z2
-> Materialize
-> Seq Scan on z1
Filter: (((a % 2) = 0) AND f_leak(b))
-(7 rows)
+(5 rows)
SET SESSION AUTHORIZATION regress_rls_carol;
SELECT * FROM z1 WHERE f_leak(b);
@@ -2278,28 +2270,24 @@ EXPLAIN (COSTS OFF) EXECUTE plancache_test;
(2 rows)
EXPLAIN (COSTS OFF) EXECUTE plancache_test2;
- QUERY PLAN
--------------------------------------------------
+ QUERY PLAN
+-----------------------------------------------------
Nested Loop
- CTE q
- -> Seq Scan on z1
- Filter: (((a % 2) = 1) AND f_leak(b))
- -> CTE Scan on q
+ -> Seq Scan on z2
-> Materialize
- -> Seq Scan on z2
-(7 rows)
+ -> Seq Scan on z1
+ Filter: (((a % 2) = 1) AND f_leak(b))
+(5 rows)
EXPLAIN (COSTS OFF) EXECUTE plancache_test3;
QUERY PLAN
-----------------------------------------------------
Nested Loop
- CTE q
- -> Seq Scan on z2
- -> CTE Scan on q
+ -> Seq Scan on z2
-> Materialize
-> Seq Scan on z1
Filter: (((a % 2) = 1) AND f_leak(b))
-(7 rows)
+(5 rows)
SET ROLE regress_rls_group2;
SELECT * FROM z1 WHERE f_leak(b);
@@ -2326,28 +2314,24 @@ EXPLAIN (COSTS OFF) EXECUTE plancache_test;
(2 rows)
EXPLAIN (COSTS OFF) EXECUTE plancache_test2;
- QUERY PLAN
--------------------------------------------------
+ QUERY PLAN
+-----------------------------------------------------
Nested Loop
- CTE q
- -> Seq Scan on z1
- Filter: (((a % 2) = 1) AND f_leak(b))
- -> CTE Scan on q
+ -> Seq Scan on z2
-> Materialize
- -> Seq Scan on z2
-(7 rows)
+ -> Seq Scan on z1
+ Filter: (((a % 2) = 1) AND f_leak(b))
+(5 rows)
EXPLAIN (COSTS OFF) EXECUTE plancache_test3;
QUERY PLAN
-----------------------------------------------------
Nested Loop
- CTE q
- -> Seq Scan on z2
- -> CTE Scan on q
+ -> Seq Scan on z2
-> Materialize
-> Seq Scan on z1
Filter: (((a % 2) = 1) AND f_leak(b))
-(7 rows)
+(5 rows)
--
-- Views should follow policy for view owner.
@@ -2854,13 +2838,11 @@ NOTICE: f_leak => 98f13708210194c475687be6106a3b84
(11 rows)
EXPLAIN (COSTS OFF) WITH cte1 AS (SELECT * FROM t1 WHERE f_leak(b)) SELECT * FROM cte1;
- QUERY PLAN
--------------------------------------------------
- CTE Scan on cte1
- CTE cte1
- -> Seq Scan on t1
- Filter: (((a % 2) = 0) AND f_leak(b))
-(4 rows)
+ QUERY PLAN
+-----------------------------------------
+ Seq Scan on t1
+ Filter: (((a % 2) = 0) AND f_leak(b))
+(2 rows)
WITH cte1 AS (UPDATE t1 SET a = a + 1 RETURNING *) SELECT * FROM cte1; --fail
ERROR: new row violates row-level security policy for table "t1"
diff --git a/src/test/regress/expected/rowtypes.out b/src/test/regress/expected/rowtypes.out
index 30053d07dff..c0df2f6333a 100644
--- a/src/test/regress/expected/rowtypes.out
+++ b/src/test/regress/expected/rowtypes.out
@@ -1041,14 +1041,11 @@ with r(a,b) as
(values (1,row(1,2)), (1,row(null,null)), (1,null),
(null,row(1,2)), (null,row(null,null)), (null,null) )
select r, r is null as isnull, r is not null as isnotnull from r;
- QUERY PLAN
-----------------------------------------------------------
- CTE Scan on r
- Output: r.*, (r.* IS NULL), (r.* IS NOT NULL)
- CTE r
- -> Values Scan on "*VALUES*"
- Output: "*VALUES*".column1, "*VALUES*".column2
-(5 rows)
+ QUERY PLAN
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ Values Scan on "*VALUES*"
+ Output: ROW("*VALUES*".column1, "*VALUES*".column2), (("*VALUES*".column1 IS NULL) AND ("*VALUES*".column2 IS NOT DISTINCT FROM NULL)), (("*VALUES*".column1 IS NOT NULL) AND ("*VALUES*".column2 IS DISTINCT FROM NULL))
+(2 rows)
with r(a,b) as
(values (1,row(1,2)), (1,row(null,null)), (1,null),
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 078129f251b..91eaf600af0 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -3135,10 +3135,8 @@ RETURNING *;
Conflict Resolution: UPDATE
Conflict Arbiter Indexes: hat_data_unique_idx
Conflict Filter: ((excluded.hat_color <> 'forbidden'::bpchar) AND (hat_data.* <> excluded.*))
- CTE data
- -> Values Scan on "*VALUES*"
- -> CTE Scan on data
-(7 rows)
+ -> Values Scan on "*VALUES*"
+(5 rows)
SELECT * FROM hat_data WHERE hat_name IN ('h8', 'h9', 'h7') ORDER BY hat_name;
hat_name | hat_color
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
@ 2018-10-03 15:57 ` David Fetter <david@fetter.org>
2018-10-04 09:22 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
1 sibling, 1 reply; 112+ messages in thread
From: David Fetter @ 2018-10-03 15:57 UTC (permalink / raw)
To: Andreas Karlsson <andreas@proxel.se>; +Cc: Thomas Munro <thomas.munro@enterprisedb.com>; Andrew Gierth <andrew@tao11.riddles.org.uk>; pgsql-hackers
On Tue, Oct 02, 2018 at 12:03:06PM +0200, Andreas Karlsson wrote:
> Hi,
>
> Here is an updated patch which adds some simple syntax for adding the
> optimization barrier. For example:
>
> WITH x AS MATERIALIZED (
> SELECT 1
> )
> SELECT * FROM x;
>
> Andreas
This is great!
Is there any meaningful distinction between "inlining," by which I
mean converting to a subquery, and predicate pushdown, which
would happen at least for a first cut, at the rewrite stage?
Best,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778
Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-03 15:57 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
@ 2018-10-04 09:22 ` Andreas Karlsson <andreas@proxel.se>
2018-10-04 18:49 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-05 00:40 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
0 siblings, 2 replies; 112+ messages in thread
From: Andreas Karlsson @ 2018-10-04 09:22 UTC (permalink / raw)
To: David Fetter <david@fetter.org>; +Cc: Thomas Munro <thomas.munro@enterprisedb.com>; Andrew Gierth <andrew@tao11.riddles.org.uk>; pgsql-hackers
On 10/03/2018 05:57 PM, David Fetter wrote:
> Is there any meaningful distinction between "inlining," by which I
> mean converting to a subquery, and predicate pushdown, which
> would happen at least for a first cut, at the rewrite stage?
Sorry, but I do not think I understand your question. The ability to
push down predicates is just one of the potential benefits from inlining.
Andreas
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-03 15:57 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-04 09:22 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
@ 2018-10-04 18:49 ` David Fetter <david@fetter.org>
1 sibling, 0 replies; 112+ messages in thread
From: David Fetter @ 2018-10-04 18:49 UTC (permalink / raw)
To: Andreas Karlsson <andreas@proxel.se>; +Cc: Thomas Munro <thomas.munro@enterprisedb.com>; Andrew Gierth <andrew@tao11.riddles.org.uk>; pgsql-hackers
On Thu, Oct 04, 2018 at 11:22:32AM +0200, Andreas Karlsson wrote:
> On 10/03/2018 05:57 PM, David Fetter wrote:
> >Is there any meaningful distinction between "inlining," by which I
> >mean converting to a subquery, and predicate pushdown, which
> >would happen at least for a first cut, at the rewrite stage?
>
> Sorry, but I do not think I understand your question. The ability to push
> down predicates is just one of the potential benefits from inlining.
Oracle appears to have such a distinction, and it appears we don't.
https://medium.com/@hakibenita/be-careful-with-cte-in-postgresql-fca5e24d2119
Best,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778
Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-03 15:57 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-04 09:22 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
@ 2018-10-05 00:40 ` Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-10-05 22:43 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
1 sibling, 1 reply; 112+ messages in thread
From: Andrew Gierth @ 2018-10-05 00:40 UTC (permalink / raw)
To: Andreas Karlsson <andreas@proxel.se>; +Cc: David Fetter <david@fetter.org>; Thomas Munro <thomas.munro@enterprisedb.com>; pgsql-hackers
>>>>> "Andreas" == Andreas Karlsson <andreas@proxel.se> writes:
> On 10/03/2018 05:57 PM, David Fetter wrote:
>> Is there any meaningful distinction between "inlining," by which I
>> mean converting to a subquery, and predicate pushdown, which
>> would happen at least for a first cut, at the rewrite stage?
Yes.
Andreas> Sorry, but I do not think I understand your question. The
Andreas> ability to push down predicates is just one of the potential
Andreas> benefits from inlining.
Consider the difference between (in the absence of CTE inlining):
-- inline subquery with no optimization barrier (qual may be pushed down)
select * from (select x from y) s where x=1;
-- inline subquery with optimization barrier (qual not pushed down)
select * from (select x from y offset 0) s where x=1;
-- CTE with materialization
with s as (select x from y) select * from s where x=1;
--
Andrew (irc:RhodiumToad)
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-03 15:57 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-04 09:22 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-05 00:40 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
@ 2018-10-05 22:43 ` David Fetter <david@fetter.org>
2018-10-06 02:48 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
0 siblings, 1 reply; 112+ messages in thread
From: David Fetter @ 2018-10-05 22:43 UTC (permalink / raw)
To: Andrew Gierth <andrew@tao11.riddles.org.uk>; +Cc: Andreas Karlsson <andreas@proxel.se>; Thomas Munro <thomas.munro@enterprisedb.com>; pgsql-hackers
On Fri, Oct 05, 2018 at 01:40:05AM +0100, Andrew Gierth wrote:
> >>>>> "Andreas" == Andreas Karlsson <andreas@proxel.se> writes:
>
> > On 10/03/2018 05:57 PM, David Fetter wrote:
> >> Is there any meaningful distinction between "inlining," by which I
> >> mean converting to a subquery, and predicate pushdown, which
> >> would happen at least for a first cut, at the rewrite stage?
>
> Yes.
>
> Andreas> Sorry, but I do not think I understand your question. The
> Andreas> ability to push down predicates is just one of the potential
> Andreas> benefits from inlining.
>
> Consider the difference between (in the absence of CTE inlining):
>
> -- inline subquery with no optimization barrier (qual may be pushed down)
> select * from (select x from y) s where x=1;
...and doesn't need to materialize all of y,
> -- inline subquery with optimization barrier (qual not pushed down)
> select * from (select x from y offset 0) s where x=1;
>
> -- CTE with materialization
> with s as (select x from y) select * from s where x=1;
while both of these do. I was interested to discover that on my
synthetic test of 10 million integers from generate_series(1,10000000)
both with and without a b-tree index on x--as expected, the index has
no effect--I consistently get stuff like this:
shackle@[local]:5432/shackle(10.5)(18539) > explain (analyze, verbose, costs on, buffers on, timing on) with s as (select x from y) select * from s where x=1;
QUERY PLAN
══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════
CTE Scan on s (cost=144247.77..369247.25 rows=50000 width=4) (actual time=0.213..2287.355 rows=1 loops=1)
Output: s.x
Filter: (s.x = 1)
Rows Removed by Filter: 9999999
Buffers: shared hit=16310 read=27938, temp written=17089
CTE s
-> Seq Scan on public.y (cost=0.00..144247.77 rows=9999977 width=4) (actual time=0.208..593.426 rows=10000000 loops=1)
Output: y.x
Buffers: shared hit=16310 read=27938
Planning time: 0.110 ms
Execution time: 2313.682 ms
(11 rows)
shackle@[local]:5432/shackle(10.5)(18539) > explain (analyze, verbose, costs on, buffers on, timing on) select * from (select x from y offset 0) s where x=1;
QUERY PLAN
════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════
Subquery Scan on s (cost=0.00..269247.48 rows=1 width=4) (actual time=0.734..1069.012 rows=1 loops=1)
Output: s.x
Filter: (s.x = 1)
Rows Removed by Filter: 9999999
Buffers: shared hit=16316 read=27932
-> Seq Scan on public.y (cost=0.00..144247.77 rows=9999977 width=4) (actual time=0.731..539.463 rows=10000000 loops=1)
Output: y.x
Buffers: shared hit=16316 read=27932
Planning time: 0.114 ms
Execution time: 1069.032 ms
(10 rows)
i.e. for this case, the CTE scan takes over 2.3x the time the simple
materialization does. Also, when I boost work_mem to 1GB (256MB wasn't
enough to avoid "temp written"), there's still a 1.8x penalty.
Best,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778
Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-03 15:57 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-04 09:22 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-05 00:40 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-10-05 22:43 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
@ 2018-10-06 02:48 ` Andrew Gierth <andrew@tao11.riddles.org.uk>
0 siblings, 0 replies; 112+ messages in thread
From: Andrew Gierth @ 2018-10-06 02:48 UTC (permalink / raw)
To: David Fetter <david@fetter.org>; +Cc: Andreas Karlsson <andreas@proxel.se>; Thomas Munro <thomas.munro@enterprisedb.com>; pgsql-hackers
>>>>> "David" == David Fetter <david@fetter.org> writes:
>> Consider the difference between (in the absence of CTE inlining):
>>
>> -- inline subquery with no optimization barrier (qual may be pushed down)
>> select * from (select x from y) s where x=1;
David> ...and doesn't need to materialize all of y,
>> -- inline subquery with optimization barrier (qual not pushed down)
>> select * from (select x from y offset 0) s where x=1;
>>
>> -- CTE with materialization
>> with s as (select x from y) select * from s where x=1;
David> while both of these do.
The non-CTE one has to _evaluate_ the whole of the "s" subquery, but it
doesn't have to actually store the result, whereas the CTE version needs
to put it all in a tuplestore and read it back.
--
Andrew (irc:RhodiumToad)
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
@ 2018-11-16 21:15 ` Tom Lane <tgl@sss.pgh.pa.us>
2018-11-16 22:41 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-11-16 23:04 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
1 sibling, 3 replies; 112+ messages in thread
From: Tom Lane @ 2018-11-16 21:15 UTC (permalink / raw)
To: Andreas Karlsson <andreas@proxel.se>; +Cc: David Fetter <david@fetter.org>; Thomas Munro <thomas.munro@enterprisedb.com>; Andrew Gierth <andrew@tao11.riddles.org.uk>; pgsql-hackers
Andreas Karlsson <andreas@proxel.se> writes:
> [ inlining-ctes-v5.patch ]
I took a little bit of a look through this. Some thoughts:
* I think it'd be a good idea if we made OFFSET/LIMIT in a CTE be an
alternate way of keeping it from being inlined. As the patch stands,
if that's the behavior you want, you have no way to express it in
a query that will also work in older servers. (I will manfully
resist suggesting that then we don't need the nonstandard syntax
at all ... oops, too late.)
* This is not the idiomatic way to declare an expression tree walker:
+static bool inline_cte_walker(Node *node, void *ctxp)
+{
+ struct inline_cte_walker_ctx *ctx = ctxp;
* I have no faith in the idea that we can skip doing a copyObject on the
inlined subquery, except maybe in the case where we know there's exactly
one reference.
* In "here's where we do the actual substitution", if we're going to
scribble on the RTE rather than make a new one, we must take pains
to zero out the RTE_CTE-specific fields so that the RTE looks the
same as if it had been a RTE_SUBQUERY all along; cf db1071d4e.
* The lack of comments about what conditions we inline under
(at subselect.c:1318) is distressing. I'm not particularly
in love with the way that inline_cte_walker is commented, either.
And dare I mention that this falsifies the intro comment for
SS_process_ctes?
* Speaking of the comments, I'm not convinced that view rewrite is
a good comparison point; I think this is more like subquery pullup.
* I wonder whether we should make range_table_walker more friendly
to the needs of this patch. The fact that it doesn't work for this usage
suggests that it might not work for others, too. I could see replacing
the QTW_EXAMINE_RTES flag with QTW_EXAMINE_RTES_BEFORE and
QTW_EXAMINE_RTES_AFTER so that callers could say which order of operations
they want (ie, visit RTE itself before or after its substructure). Then
we could get rid of the double traversal of the RTE list.
* I think a large fraction of the regression test cases that this
changes are actually broken by the patch, in the sense that we meant
to test the behavior with a CTE and now we're not getting that.
So we'd need to add MATERIALIZED in many more places than this has
done. Somebody else (Stephen?) would need to opine on whether that's
true for the CTEs in rowsecurity.sql, but it's definitely true for
the one in rowtypes.sql, where the point is to test what happens
with a whole-row Var.
* Which will mean we need some new test cases showing that this patch does
anything. It'd be a good idea to add a test case showing that this gets
things right for conflicting CTE names at different levels, eg
explain verbose
with x as (select 1 as y)
select * from (with x as (select 2 as y) select * from x) ss;
* ruleutils.c needs adjustments for the new syntax, if we keep that.
* And of course the documentation needs much more work than this.
regards, tom lane
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
@ 2018-11-16 22:41 ` David Fetter <david@fetter.org>
2 siblings, 0 replies; 112+ messages in thread
From: David Fetter @ 2018-11-16 22:41 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Andreas Karlsson <andreas@proxel.se>; Thomas Munro <thomas.munro@enterprisedb.com>; Andrew Gierth <andrew@tao11.riddles.org.uk>; pgsql-hackers
On Fri, Nov 16, 2018 at 04:15:10PM -0500, Tom Lane wrote:
> Andreas Karlsson <andreas@proxel.se> writes:
> > [ inlining-ctes-v5.patch ]
>
> I took a little bit of a look through this. Some thoughts:
>
> * I think it'd be a good idea if we made OFFSET/LIMIT in a CTE be an
> alternate way of keeping it from being inlined. As the patch stands,
> if that's the behavior you want, you have no way to express it in
> a query that will also work in older servers. (I will manfully
> resist suggesting that then we don't need the nonstandard syntax
> at all ... oops, too late.)
If we're on board with exposing pilot error, we could decide not to
implement the nonstandard WITH syntax. One type of pilot error this
would expose is a situation where:
- A UDF has side effects, but is declared IMMUTABLE
- A WITH clause calls it in order to get those side effects on the
entire result set
Best,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778
Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
@ 2018-11-16 23:04 ` Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-11-16 23:24 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-11-17 02:12 ` Re: Early WIP/PoC for inlining CTEs Stephen Frost <sfrost@snowman.net>
2 siblings, 2 replies; 112+ messages in thread
From: Andrew Gierth @ 2018-11-16 23:04 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Andreas Karlsson <andreas@proxel.se>; David Fetter <david@fetter.org>; Thomas Munro <thomas.munro@enterprisedb.com>; pgsql-hackers
>>>>> "Tom" == Tom Lane <tgl@sss.pgh.pa.us> writes:
>> [ inlining-ctes-v5.patch ]
Tom> I took a little bit of a look through this. Some thoughts:
Tom> * I think it'd be a good idea if we made OFFSET/LIMIT in a CTE be
Tom> an alternate way of keeping it from being inlined. As the patch
Tom> stands, if that's the behavior you want, you have no way to
Tom> express it in a query that will also work in older servers. (I
Tom> will manfully resist suggesting that then we don't need the
Tom> nonstandard syntax at all ... oops, too late.)
I think this is the wrong approach, because you may want the
optimization-barrier effects of OFFSET/LIMIT _without_ the actual
materialization - there is no need to force a query like
with d as (select stuff from bigtable offset 1) select * from d;
to push all the data through an (on-disk) tuplestore.
Tom> * I have no faith in the idea that we can skip doing a copyObject
Tom> on the inlined subquery, except maybe in the case where we know
Tom> there's exactly one reference.
The code doesn't do a copyObject on the query if there are no level
changes because everywhere else just does another copyObject on it as
the first processing step (cf. set_subquery_pathlist and the various
functions called from pull_up_subqueries).
Tom> * In "here's where we do the actual substitution", if we're going
Tom> to scribble on the RTE rather than make a new one, we must take
Tom> pains to zero out the RTE_CTE-specific fields so that the RTE
Tom> looks the same as if it had been a RTE_SUBQUERY all along; cf
Tom> db1071d4e.
Yes, this needs fixing. (This code predates that commit.)
Tom> * Speaking of the comments, I'm not convinced that view rewrite is
Tom> a good comparison point; I think this is more like subquery
Tom> pullup.
It's not really like subquery pullup because we actually do go on to do
subquery pullup _after_ inlining CTEs.
Tom> * I wonder whether we should make range_table_walker more friendly
Tom> to the needs of this patch. The fact that it doesn't work for this
Tom> usage suggests that it might not work for others, too. I could see
Tom> replacing the QTW_EXAMINE_RTES flag with QTW_EXAMINE_RTES_BEFORE
Tom> and QTW_EXAMINE_RTES_AFTER so that callers could say which order
Tom> of operations they want (ie, visit RTE itself before or after its
Tom> substructure). Then we could get rid of the double traversal of
Tom> the RTE list.
Sure, why not.
--
Andrew.
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-11-16 23:04 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
@ 2018-11-16 23:24 ` Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 0 replies; 112+ messages in thread
From: Tom Lane @ 2018-11-16 23:24 UTC (permalink / raw)
To: Andrew Gierth <andrew@tao11.riddles.org.uk>; +Cc: Andreas Karlsson <andreas@proxel.se>; David Fetter <david@fetter.org>; Thomas Munro <thomas.munro@enterprisedb.com>; pgsql-hackers
Andrew Gierth <andrew@tao11.riddles.org.uk> writes:
> "Tom" == Tom Lane <tgl@sss.pgh.pa.us> writes:
> Tom> * I have no faith in the idea that we can skip doing a copyObject
> Tom> on the inlined subquery, except maybe in the case where we know
> Tom> there's exactly one reference.
> The code doesn't do a copyObject on the query if there are no level
> changes because everywhere else just does another copyObject on it as
> the first processing step (cf. set_subquery_pathlist and the various
> functions called from pull_up_subqueries).
Perhaps it's safe today, but is that likely to remain true? We've had
enough pain in the past from multiply-linked parse subtrees that I am
not eager to introduce another case, especially not here where there's
absolutely no evidence that it'd provide a meaningful performance
improvement.
> Tom> * Speaking of the comments, I'm not convinced that view rewrite is
> Tom> a good comparison point; I think this is more like subquery
> Tom> pullup.
> It's not really like subquery pullup because we actually do go on to do
> subquery pullup _after_ inlining CTEs.
Subquery pullup can happen across multiple levels, too.
regards, tom lane
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-11-16 23:04 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
@ 2018-11-17 02:12 ` Stephen Frost <sfrost@snowman.net>
2018-11-19 03:38 ` Re: Early WIP/PoC for inlining CTEs Craig Ringer <craig@2ndquadrant.com>
1 sibling, 1 reply; 112+ messages in thread
From: Stephen Frost @ 2018-11-17 02:12 UTC (permalink / raw)
To: Andrew Gierth <andrew@tao11.riddles.org.uk>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Andreas Karlsson <andreas@proxel.se>; David Fetter <david@fetter.org>; Thomas Munro <thomas.munro@enterprisedb.com>; pgsql-hackers
Greetings,
* Andrew Gierth (andrew@tao11.riddles.org.uk) wrote:
> >>>>> "Tom" == Tom Lane <tgl@sss.pgh.pa.us> writes:
>
> >> [ inlining-ctes-v5.patch ]
>
> Tom> I took a little bit of a look through this. Some thoughts:
>
> Tom> * I think it'd be a good idea if we made OFFSET/LIMIT in a CTE be
> Tom> an alternate way of keeping it from being inlined. As the patch
> Tom> stands, if that's the behavior you want, you have no way to
> Tom> express it in a query that will also work in older servers. (I
> Tom> will manfully resist suggesting that then we don't need the
> Tom> nonstandard syntax at all ... oops, too late.)
>
> I think this is the wrong approach, because you may want the
> optimization-barrier effects of OFFSET/LIMIT _without_ the actual
> materialization - there is no need to force a query like
>
> with d as (select stuff from bigtable offset 1) select * from d;
>
> to push all the data through an (on-disk) tuplestore.
Agreed, there's going to be cases where you want the CTE to be inlined
even with OFFSET/LIMIT. Let's please not cater to the crowd who
happened to know that they could hack around with OFFSET/LIMIT to make
something not be inlined when it comes to the question of if the CTE
should be inlined or not. That's the same issue we were argueing around
when discussing if we should allow parallel array_agg, imv.
Particularly since, with CTEs anyway, we never inlined them, so the
whole OFFSET/LIMIT thing doesn't really make any sense- today, if you
wrote a CTE, you wouldn't bother with OFFSET/LIMIT because you knew it
wasn't going to be inlined, that entire line of thinking is for
subqueries, not CTEs. If you're going to force people to change their
CTEs to require that they not be inlined, let's not pick a method which
makes it ambiguous and makes us have to ask "do they really want this
limit/offset, or did they just want to make the CTE not be inlined...?"
Thanks!
Stephen
Attachments:
[application/pgp-signature] signature.asc (818B, ../../20181117021239.GI3415@tamriel.snowman.net/2-signature.asc)
download
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-11-16 23:04 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-11-17 02:12 ` Re: Early WIP/PoC for inlining CTEs Stephen Frost <sfrost@snowman.net>
@ 2018-11-19 03:38 ` Craig Ringer <craig@2ndquadrant.com>
0 siblings, 0 replies; 112+ messages in thread
From: Craig Ringer @ 2018-11-19 03:38 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Andrew Gierth <andrew@tao11.riddles.org.uk>; Tom Lane <tgl@sss.pgh.pa.us>; Andreas Karlsson <andreas@proxel.se>; David Fetter <david@fetter.org>; Thomas Munro <thomas.munro@enterprisedb.com>; pgsql-hackers
On Sat, 17 Nov 2018 at 10:12, Stephen Frost <sfrost@snowman.net> wrote:
> Greetings,
>
> * Andrew Gierth (andrew@tao11.riddles.org.uk) wrote:
> > >>>>> "Tom" == Tom Lane <tgl@sss.pgh.pa.us> writes:
> >
> > >> [ inlining-ctes-v5.patch ]
> >
> > Tom> I took a little bit of a look through this. Some thoughts:
> >
> > Tom> * I think it'd be a good idea if we made OFFSET/LIMIT in a CTE be
> > Tom> an alternate way of keeping it from being inlined. As the patch
> > Tom> stands, if that's the behavior you want, you have no way to
> > Tom> express it in a query that will also work in older servers. (I
> > Tom> will manfully resist suggesting that then we don't need the
> > Tom> nonstandard syntax at all ... oops, too late.)
> >
> > I think this is the wrong approach, because you may want the
> > optimization-barrier effects of OFFSET/LIMIT _without_ the actual
> > materialization - there is no need to force a query like
> >
> > with d as (select stuff from bigtable offset 1) select * from d;
> >
> > to push all the data through an (on-disk) tuplestore.
>
> Agreed, there's going to be cases where you want the CTE to be inlined
> even with OFFSET/LIMIT. Let's please not cater to the crowd who
> happened to know that they could hack around with OFFSET/LIMIT to make
> something not be inlined when it comes to the question of if the CTE
> should be inlined or not. That's the same issue we were argueing around
> when discussing if we should allow parallel array_agg, imv.
>
> Particularly since, with CTEs anyway, we never inlined them, so the
> whole OFFSET/LIMIT thing doesn't really make any sense- today, if you
> wrote a CTE, you wouldn't bother with OFFSET/LIMIT because you knew it
> wasn't going to be inlined, that entire line of thinking is for
> subqueries, not CTEs. If you're going to force people to change their
> CTEs to require that they not be inlined, let's not pick a method which
> makes it ambiguous and makes us have to ask "do they really want this
> limit/offset, or did they just want to make the CTE not be inlined...?"
>
>
To satisfy Tom's understandable desire to let people write queries that
behave the same on old and new versions, can we get away with back-patching
the MATERIALIZED parser enhancement as a no-op in point releases?
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
@ 2018-12-31 23:46 ` Andreas Karlsson <andreas@proxel.se>
2019-01-01 00:42 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2 siblings, 2 replies; 112+ messages in thread
From: Andreas Karlsson @ 2018-12-31 23:46 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: David Fetter <david@fetter.org>; Thomas Munro <thomas.munro@enterprisedb.com>; Andrew Gierth <andrew@tao11.riddles.org.uk>; pgsql-hackers
On 11/16/18 10:15 PM, Tom Lane wrote:
> I took a little bit of a look through this. Some thoughts:
Thanks for the review! I have decided to pick up this patch and work on
it since nothing has happened in a while. Here is a new version with
most of the feedback fixed.
> * This is not the idiomatic way to declare an expression tree walker:
>
> +static bool inline_cte_walker(Node *node, void *ctxp)
> +{
> + struct inline_cte_walker_ctx *ctx = ctxp;
>
> * I have no faith in the idea that we can skip doing a copyObject on the
> inlined subquery, except maybe in the case where we know there's exactly
> one reference.
>
> * In "here's where we do the actual substitution", if we're going to
> scribble on the RTE rather than make a new one, we must take pains
> to zero out the RTE_CTE-specific fields so that the RTE looks the
> same as if it had been a RTE_SUBQUERY all along; cf db1071d4e.
>
> * The lack of comments about what conditions we inline under
> (at subselect.c:1318) is distressing. I'm not particularly
> in love with the way that inline_cte_walker is commented, either.
> And dare I mention that this falsifies the intro comment for
> SS_process_ctes?
>
> * Speaking of the comments, I'm not convinced that view rewrite is
> a good comparison point; I think this is more like subquery pullup.
I believe I have fixed these except for the comment on the conditions
for when we inline.
Andrew Gierth: Why did you chose to not inline on FOR UPDATE but inline
volatile functions? I feel that this might be inconsistent since in both
cases the query in the CTE can change behavior if the planner pushes a
WHERE clause into the subquery, but maybe I am missing something.
> * I wonder whether we should make range_table_walker more friendly
> to the needs of this patch. The fact that it doesn't work for this usage
> suggests that it might not work for others, too. I could see replacing
> the QTW_EXAMINE_RTES flag with QTW_EXAMINE_RTES_BEFORE and
> QTW_EXAMINE_RTES_AFTER so that callers could say which order of operations
> they want (ie, visit RTE itself before or after its substructure). Then
> we could get rid of the double traversal of the RTE list.
I did as suggested and the code is now much cleaner, but I feel while
RTE walking business would become cleaner if we could change from having
the range table walker yield both RTEs and the contents of the RTEs to
having it just yeild the RTEs and then the walker callback can call
expression_tree_walker() with the RTE so RTEs are treated like any other
node in the tree.
I might look into how big impact such a change would have and if it is
worth the churn.
> * I think a large fraction of the regression test cases that this
> changes are actually broken by the patch, in the sense that we meant
> to test the behavior with a CTE and now we're not getting that.
> So we'd need to add MATERIALIZED in many more places than this has
> done. Somebody else (Stephen?) would need to opine on whether that's
> true for the CTEs in rowsecurity.sql, but it's definitely true for
> the one in rowtypes.sql, where the point is to test what happens
> with a whole-row Var.
Agreed, fixed.
> * Which will mean we need some new test cases showing that this patch does
> anything. It'd be a good idea to add a test case showing that this gets
> things right for conflicting CTE names at different levels, eg
>
> explain verbose
> with x as (select 1 as y)
> select * from (with x as (select 2 as y) select * from x) ss;
Added this test case, but more are needed. Any suggestion for what file
these tests belong (right now I just added it to subselect.sql)? Or
should I add a new called cte.sql?
> * ruleutils.c needs adjustments for the new syntax, if we keep that.
Thanks, fixed!
> * And of course the documentation needs much more work than this.
Yeah, I was waiting for there to be more agreement on when CTEs should
be inlined, but maybe I should start writing anyway.
Andreas
Attachments:
[text/x-patch] inlining-ctes-v6.patch (24.3K, ../../d4daf011-01cf-6bf9-3f15-85ae1572e909@proxel.se/2-inlining-ctes-v6.patch)
download | inline diff:
diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out
index bb92d9d37a..8c26dd1f26 100644
--- a/contrib/postgres_fdw/expected/postgres_fdw.out
+++ b/contrib/postgres_fdw/expected/postgres_fdw.out
@@ -1888,7 +1888,7 @@ SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) ORDER BY t1.c3, t
-- join in CTE
EXPLAIN (VERBOSE, COSTS OFF)
-WITH t (c1_1, c1_3, c2_1) AS (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
+WITH t (c1_1, c1_3, c2_1) AS MATERIALIZED (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
QUERY PLAN
-------------------------------------------------------------------------------------------------------------------------------------
Limit
@@ -1905,7 +1905,7 @@ WITH t (c1_1, c1_3, c2_1) AS (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2
Output: t.c1_1, t.c2_1, t.c1_3
(12 rows)
-WITH t (c1_1, c1_3, c2_1) AS (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
+WITH t (c1_1, c1_3, c2_1) AS MATERIALIZED (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
c1_1 | c2_1
------+------
101 | 101
diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql
index f438165650..56602a164c 100644
--- a/contrib/postgres_fdw/sql/postgres_fdw.sql
+++ b/contrib/postgres_fdw/sql/postgres_fdw.sql
@@ -493,8 +493,8 @@ SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) ORDER BY t1.c3, t
SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) ORDER BY t1.c3, t1.c1 OFFSET 100 LIMIT 10 FOR SHARE;
-- join in CTE
EXPLAIN (VERBOSE, COSTS OFF)
-WITH t (c1_1, c1_3, c2_1) AS (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
-WITH t (c1_1, c1_3, c2_1) AS (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
+WITH t (c1_1, c1_3, c2_1) AS MATERIALIZED (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
+WITH t (c1_1, c1_3, c2_1) AS MATERIALIZED (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
-- ctid with whole-row reference
EXPLAIN (VERBOSE, COSTS OFF)
SELECT t1.ctid, t1, t2, t1.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) ORDER BY t1.c3, t1.c1 OFFSET 100 LIMIT 10;
diff --git a/doc/src/sgml/queries.sgml b/doc/src/sgml/queries.sgml
index 88bc189646..fe47ba0d1f 100644
--- a/doc/src/sgml/queries.sgml
+++ b/doc/src/sgml/queries.sgml
@@ -2199,6 +2199,8 @@ SELECT n FROM t LIMIT 100;
</para>
<para>
+ TODO: Update this for inlinding and <literal>MATERILIZED</literal>.
+
A useful property of <literal>WITH</literal> queries is that they are evaluated
only once per execution of the parent query, even if they are referred to
more than once by the parent query or sibling <literal>WITH</literal> queries.
diff --git a/doc/src/sgml/ref/select.sgml b/doc/src/sgml/ref/select.sgml
index 4db8142afa..e344b7e49a 100644
--- a/doc/src/sgml/ref/select.sgml
+++ b/doc/src/sgml/ref/select.sgml
@@ -72,7 +72,7 @@ SELECT [ ALL | DISTINCT [ ON ( <replaceable class="parameter">expression</replac
<phrase>and <replaceable class="parameter">with_query</replaceable> is:</phrase>
- <replaceable class="parameter">with_query_name</replaceable> [ ( <replaceable class="parameter">column_name</replaceable> [, ...] ) ] AS ( <replaceable class="parameter">select</replaceable> | <replaceable class="parameter">values</replaceable> | <replaceable class="parameter">insert</replaceable> | <replaceable class="parameter">update</replaceable> | <replaceable class="parameter">delete</replaceable> )
+ <replaceable class="parameter">with_query_name</replaceable> [ ( <replaceable class="parameter">column_name</replaceable> [, ...] ) ] AS [ MATERIALIZED ] ( <replaceable class="parameter">select</replaceable> | <replaceable class="parameter">values</replaceable> | <replaceable class="parameter">insert</replaceable> | <replaceable class="parameter">update</replaceable> | <replaceable class="parameter">delete</replaceable> )
TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ]
</synopsis>
@@ -290,6 +290,10 @@ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ]
row, the results are unspecified.
</para>
+ <para>
+ TODO: Write something about <literal>MATERIALIZED</literal>.
+ </para>
+
<para>
See <xref linkend="queries-with"/> for additional information.
</para>
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 144cd7a047..fad7a0b3c2 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2539,6 +2539,7 @@ _copyCommonTableExpr(const CommonTableExpr *from)
COPY_NODE_FIELD(ctequery);
COPY_LOCATION_FIELD(location);
COPY_SCALAR_FIELD(cterecursive);
+ COPY_SCALAR_FIELD(ctematerialized);
COPY_SCALAR_FIELD(cterefcount);
COPY_NODE_FIELD(ctecolnames);
COPY_NODE_FIELD(ctecoltypes);
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index c4579fbcc6..0f9aa03ac5 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2793,6 +2793,7 @@ _equalCommonTableExpr(const CommonTableExpr *a, const CommonTableExpr *b)
COMPARE_NODE_FIELD(ctequery);
COMPARE_LOCATION_FIELD(location);
COMPARE_SCALAR_FIELD(cterecursive);
+ COMPARE_SCALAR_FIELD(ctematerialized);
COMPARE_SCALAR_FIELD(cterefcount);
COMPARE_NODE_FIELD(ctecolnames);
COMPARE_NODE_FIELD(ctecoltypes);
diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c
index a813e38791..9497cb67cf 100644
--- a/src/backend/nodes/nodeFuncs.c
+++ b/src/backend/nodes/nodeFuncs.c
@@ -2314,8 +2314,7 @@ range_table_walker(List *rtable,
{
RangeTblEntry *rte = (RangeTblEntry *) lfirst(rt);
- /* For historical reasons, visiting RTEs is not the default */
- if (flags & QTW_EXAMINE_RTES)
+ if (flags & QTW_EXAMINE_RTES_BEFORE)
if (walker(rte, context))
return true;
@@ -2353,6 +2352,10 @@ range_table_walker(List *rtable,
break;
}
+ if (flags & QTW_EXAMINE_RTES_AFTER)
+ if (walker(rte, context))
+ return true;
+
if (walker(rte->securityQuals, context))
return true;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 2dd8ed3a46..e9b70856d3 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2983,6 +2983,7 @@ _outCommonTableExpr(StringInfo str, const CommonTableExpr *node)
WRITE_NODE_FIELD(ctequery);
WRITE_LOCATION_FIELD(location);
WRITE_BOOL_FIELD(cterecursive);
+ WRITE_BOOL_FIELD(ctematerialized);
WRITE_INT_FIELD(cterefcount);
WRITE_NODE_FIELD(ctecolnames);
WRITE_NODE_FIELD(ctecoltypes);
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index fa27f37d6f..a7054fa044 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -421,6 +421,7 @@ _readCommonTableExpr(void)
READ_NODE_FIELD(ctequery);
READ_LOCATION_FIELD(location);
READ_BOOL_FIELD(cterecursive);
+ READ_BOOL_FIELD(ctematerialized);
READ_INT_FIELD(cterefcount);
READ_NODE_FIELD(ctecolnames);
READ_NODE_FIELD(ctecoltypes);
diff --git a/src/backend/optimizer/plan/setrefs.c b/src/backend/optimizer/plan/setrefs.c
index 920a99d0d9..0fed2f07c7 100644
--- a/src/backend/optimizer/plan/setrefs.c
+++ b/src/backend/optimizer/plan/setrefs.c
@@ -340,7 +340,7 @@ flatten_unplanned_rtes(PlannerGlobal *glob, RangeTblEntry *rte)
(void) query_tree_walker(rte->subquery,
flatten_rtes_walker,
(void *) glob,
- QTW_EXAMINE_RTES);
+ QTW_EXAMINE_RTES_BEFORE);
}
static bool
@@ -363,7 +363,7 @@ flatten_rtes_walker(Node *node, PlannerGlobal *glob)
return query_tree_walker((Query *) node,
flatten_rtes_walker,
(void *) glob,
- QTW_EXAMINE_RTES);
+ QTW_EXAMINE_RTES_BEFORE);
}
return expression_tree_walker(node, flatten_rtes_walker,
(void *) glob);
diff --git a/src/backend/optimizer/plan/subselect.c b/src/backend/optimizer/plan/subselect.c
index 83008d7661..eebc968fde 100644
--- a/src/backend/optimizer/plan/subselect.c
+++ b/src/backend/optimizer/plan/subselect.c
@@ -53,6 +53,14 @@ typedef struct finalize_primnode_context
Bitmapset *paramids; /* Non-local PARAM_EXEC paramids found */
} finalize_primnode_context;
+typedef struct inline_cte_walker_context
+{
+ const char *ctename;
+ int levelsup;
+ int refcount;
+ Query *ctequery;
+ CommonTableExpr *cte;
+} inline_cte_walker_context;
static Node *build_subplan(PlannerInfo *root, Plan *plan, PlannerInfo *subroot,
List *plan_params,
@@ -1145,14 +1153,93 @@ hash_ok_operator(OpExpr *expr)
}
}
+/*
+ * inline_cte_walker: recursively walk the query and inline CTE
+ *
+ * While walking the expression tree with take care to examine the RTEs and
+ * potentially replace a CTE RTE with a subquery RTE after recursing into the
+ * RTE to avoid recursing into the newly inlined CTE.
+ */
+static bool inline_cte_walker(Node *node, inline_cte_walker_context *context)
+{
+ if (!node)
+ return false;
+
+ if (IsA(node, Query))
+ {
+ Query *query = castNode(Query, node);
+
+ context->levelsup++;
+
+ /*
+ * Visit the actual RTE nodes after their contents so we can modify
+ * the RTE node without having to visit the newly inlined contents.
+ */
+ query_tree_walker(query, inline_cte_walker, context,
+ QTW_EXAMINE_RTES_AFTER);
+
+ context->levelsup--;
+
+ return false;
+ }
+ else if (IsA(node, RangeTblEntry))
+ {
+ RangeTblEntry *rte = castNode(RangeTblEntry, node);
+
+ if (rte->rtekind == RTE_CTE &&
+ strcmp(rte->ctename, context->ctename) == 0 &&
+ rte->ctelevelsup == context->levelsup)
+ {
+ Query *newquery = copyObject(context->ctequery);
+
+ /* Preserve outer references, for example to other CTEs */
+ if (context->levelsup > 0)
+ IncrementVarSublevelsUp((Node *) newquery, context->levelsup, 1);
+
+ rte->rtekind = RTE_SUBQUERY;
+ rte->subquery = newquery;
+ rte->security_barrier = false;
+
+ /* Zero out CTE specific fields */
+ rte->ctename = NULL;
+ rte->ctelevelsup = 0;
+ rte->self_reference = false;
+ rte->coltypes = NIL;
+ rte->coltypmods = NIL;
+ rte->colcollations = NIL;
+
+ context->refcount--;
+ }
+
+ return false;
+ }
+
+ return expression_tree_walker(node, inline_cte_walker, context);
+}
+
+/*
+ * inline_cte: replace all refernces to CTE with subqueries
+ */
+static void inline_cte(PlannerInfo *root, CommonTableExpr *cte)
+{
+ struct inline_cte_walker_context context;
+ context.ctequery = castNode(Query, cte->ctequery);
+ context.ctename = cte->ctename;
+ context.refcount = cte->cterefcount;
+ context.levelsup = -1;
+ context.cte = cte;
+ inline_cte_walker((Node *) root->parse, &context);
+ /* we must replace all references */
+ Assert(context.refcount == 0);
+}
/*
* SS_process_ctes: process a query's WITH list
*
- * We plan each interesting WITH item and convert it to an initplan.
- * A side effect is to fill in root->cte_plan_ids with a list that
- * parallels root->parse->cteList and provides the subplan ID for
- * each CTE's initplan.
+ * We plan each interesting WITH item and either convert it to an initplan, or
+ * if allowed inline it as a subquery. A side effect is to fill in
+ * root->cte_plan_ids with a list that parallels root->parse->cteList and
+ * provides the subplan ID for each CTE's initplan, or a dummy ID when inlined.
*/
void
SS_process_ctes(PlannerInfo *root)
@@ -1183,6 +1270,24 @@ SS_process_ctes(PlannerInfo *root)
continue;
}
+ /*
+ * Consider inlining the CTE rather than planning it separately.
+ *
+ * TODO: Check with Gierth why rowMarks should not be inlined.
+ */
+ if (cmdType == CMD_SELECT &&
+ !cte->ctematerialized &&
+ !cte->cterecursive &&
+ !((cte->cterefcount > 1) && contain_volatile_functions(cte->ctequery)) &&
+ castNode(Query, cte->ctequery)->rowMarks == NIL)
+ {
+ inline_cte(root, cte);
+
+ /* Make a dummy entry in cte_plan_ids */
+ root->cte_plan_ids = lappend_int(root->cte_plan_ids, -1);
+ continue;
+ }
+
/*
* Copy the source Query node. Probably not necessary, but let's keep
* this similar to make_subplan.
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 5bcaf42205..0cef2737d5 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -518,7 +518,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
Bit ConstBit BitWithLength BitWithoutLength
%type <str> character
%type <str> extract_arg
-%type <boolean> opt_varying opt_timezone opt_no_inherit
+%type <boolean> opt_varying opt_timezone opt_no_inherit opt_materialized
%type <ival> Iconst SignedIconst
%type <str> Sconst comment_text notify_payload
@@ -11368,17 +11368,23 @@ cte_list:
| cte_list ',' common_table_expr { $$ = lappend($1, $3); }
;
-common_table_expr: name opt_name_list AS '(' PreparableStmt ')'
+common_table_expr: name opt_name_list AS opt_materialized '(' PreparableStmt ')'
{
CommonTableExpr *n = makeNode(CommonTableExpr);
n->ctename = $1;
n->aliascolnames = $2;
- n->ctequery = $5;
+ n->ctequery = $6;
+ n->ctematerialized = $4;
n->location = @1;
$$ = (Node *) n;
}
;
+opt_materialized:
+ MATERIALIZED { $$ = true; }
+ | /*EMPTY*/ { $$ = false; }
+ ;
+
opt_with_clause:
with_clause { $$ = $1; }
| /*EMPTY*/ { $$ = NULL; }
diff --git a/src/backend/rewrite/rewriteManip.c b/src/backend/rewrite/rewriteManip.c
index f1f4212b5d..5a71d691ce 100644
--- a/src/backend/rewrite/rewriteManip.c
+++ b/src/backend/rewrite/rewriteManip.c
@@ -761,7 +761,7 @@ IncrementVarSublevelsUp_walker(Node *node,
result = query_tree_walker((Query *) node,
IncrementVarSublevelsUp_walker,
(void *) context,
- QTW_EXAMINE_RTES);
+ QTW_EXAMINE_RTES_BEFORE);
context->min_sublevels_up--;
return result;
}
@@ -785,7 +785,7 @@ IncrementVarSublevelsUp(Node *node, int delta_sublevels_up,
query_or_expression_tree_walker(node,
IncrementVarSublevelsUp_walker,
(void *) &context,
- QTW_EXAMINE_RTES);
+ QTW_EXAMINE_RTES_BEFORE);
}
/*
@@ -804,7 +804,7 @@ IncrementVarSublevelsUp_rtable(List *rtable, int delta_sublevels_up,
range_table_walker(rtable,
IncrementVarSublevelsUp_walker,
(void *) &context,
- QTW_EXAMINE_RTES);
+ QTW_EXAMINE_RTES_BEFORE);
}
diff --git a/src/backend/utils/adt/ruleutils.c b/src/backend/utils/adt/ruleutils.c
index 4857caecaa..b4030a111e 100644
--- a/src/backend/utils/adt/ruleutils.c
+++ b/src/backend/utils/adt/ruleutils.c
@@ -5165,7 +5165,10 @@ get_with_clause(Query *query, deparse_context *context)
}
appendStringInfoChar(buf, ')');
}
- appendStringInfoString(buf, " AS (");
+ appendStringInfoString(buf, " AS ");
+ if (cte->ctematerialized)
+ appendStringInfoString(buf, "MATERIALIZED ");
+ appendStringInfoChar(buf, '(');
if (PRETTY_INDENT(context))
appendContextKeyword(context, "", 0, 0, 0);
get_query_def((Query *) cte->ctequery, buf, context->namespaces, NULL,
diff --git a/src/include/nodes/nodeFuncs.h b/src/include/nodes/nodeFuncs.h
index 849f34d2a8..d4783cd61f 100644
--- a/src/include/nodes/nodeFuncs.h
+++ b/src/include/nodes/nodeFuncs.h
@@ -22,8 +22,10 @@
#define QTW_IGNORE_RC_SUBQUERIES 0x03 /* both of above */
#define QTW_IGNORE_JOINALIASES 0x04 /* JOIN alias var lists */
#define QTW_IGNORE_RANGE_TABLE 0x08 /* skip rangetable entirely */
-#define QTW_EXAMINE_RTES 0x10 /* examine RTEs */
+#define QTW_EXAMINE_RTES_BEFORE 0x10 /* examine RTE nodes before their contents */
#define QTW_DONT_COPY_QUERY 0x20 /* do not copy top Query */
+#define QTW_EXAMINE_RTES_AFTER 0x40 /* examine RTE nodes after their contents */
+
/* callback function for check_functions_in_node */
typedef bool (*check_function_callback) (Oid func_id, void *context);
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index d3dd3d0339..7d4fbffcbf 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -1409,6 +1409,7 @@ typedef struct CommonTableExpr
int location; /* token location, or -1 if unknown */
/* These fields are set during parse analysis: */
bool cterecursive; /* is this CTE actually recursive? */
+ bool ctematerialized; /* is this an optimization fence? */
int cterefcount; /* number of RTEs referencing this CTE
* (excluding internal self-references) */
List *ctecolnames; /* list of output column names */
diff --git a/src/test/regress/expected/rowsecurity.out b/src/test/regress/expected/rowsecurity.out
index 1d12b01068..1a0932d863 100644
--- a/src/test/regress/expected/rowsecurity.out
+++ b/src/test/regress/expected/rowsecurity.out
@@ -2179,7 +2179,7 @@ EXPLAIN (COSTS OFF) EXECUTE plancache_test;
Filter: (((a % 2) = 0) AND f_leak(b))
(2 rows)
-PREPARE plancache_test2 AS WITH q AS (SELECT * FROM z1 WHERE f_leak(b)) SELECT * FROM q,z2;
+PREPARE plancache_test2 AS WITH q AS MATERIALIZED (SELECT * FROM z1 WHERE f_leak(b)) SELECT * FROM q,z2;
EXPLAIN (COSTS OFF) EXECUTE plancache_test2;
QUERY PLAN
-------------------------------------------------
@@ -2192,7 +2192,7 @@ EXPLAIN (COSTS OFF) EXECUTE plancache_test2;
-> Seq Scan on z2
(7 rows)
-PREPARE plancache_test3 AS WITH q AS (SELECT * FROM z2) SELECT * FROM q,z1 WHERE f_leak(z1.b);
+PREPARE plancache_test3 AS WITH q AS MATERIALIZED (SELECT * FROM z2) SELECT * FROM q,z1 WHERE f_leak(z1.b);
EXPLAIN (COSTS OFF) EXECUTE plancache_test3;
QUERY PLAN
-----------------------------------------------------
@@ -2853,7 +2853,7 @@ NOTICE: f_leak => 98f13708210194c475687be6106a3b84
20 | 98f13708210194c475687be6106a3b84
(11 rows)
-EXPLAIN (COSTS OFF) WITH cte1 AS (SELECT * FROM t1 WHERE f_leak(b)) SELECT * FROM cte1;
+EXPLAIN (COSTS OFF) WITH cte1 AS MATERIALIZED (SELECT * FROM t1 WHERE f_leak(b)) SELECT * FROM cte1;
QUERY PLAN
-------------------------------------------------
CTE Scan on cte1
diff --git a/src/test/regress/expected/rowtypes.out b/src/test/regress/expected/rowtypes.out
index d6a1a3331e..14757b40f6 100644
--- a/src/test/regress/expected/rowtypes.out
+++ b/src/test/regress/expected/rowtypes.out
@@ -1037,7 +1037,7 @@ from (values (1,row(1,2)), (1,row(null,null)), (1,null),
(6 rows)
explain (verbose, costs off)
-with r(a,b) as
+with r(a,b) as materialized
(values (1,row(1,2)), (1,row(null,null)), (1,null),
(null,row(1,2)), (null,row(null,null)), (null,null) )
select r, r is null as isnull, r is not null as isnotnull from r;
@@ -1050,7 +1050,7 @@ select r, r is null as isnull, r is not null as isnotnull from r;
Output: "*VALUES*".column1, "*VALUES*".column2
(5 rows)
-with r(a,b) as
+with r(a,b) as materialized
(values (1,row(1,2)), (1,row(null,null)), (1,null),
(null,row(1,2)), (null,row(null,null)), (null,null) )
select r, r is null as isnull, r is not null as isnotnull from r;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index e384cd2279..32dfab94b2 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -3108,7 +3108,7 @@ explain (costs off) INSERT INTO hats VALUES ('h8', 'forbidden') RETURNING *;
(5 rows)
-- ensure upserting into a rule, with a CTE (different offsets!) works
-WITH data(hat_name, hat_color) AS (
+WITH data(hat_name, hat_color) AS MATERIALIZED (
VALUES ('h8', 'green'),
('h9', 'blue'),
('h7', 'forbidden')
@@ -3122,7 +3122,7 @@ RETURNING *;
h9 | blue
(2 rows)
-EXPLAIN (costs off) WITH data(hat_name, hat_color) AS (
+EXPLAIN (costs off) WITH data(hat_name, hat_color) AS MATERIALIZED (
VALUES ('h8', 'green'),
('h9', 'blue'),
('h7', 'forbidden')
diff --git a/src/test/regress/expected/subselect.out b/src/test/regress/expected/subselect.out
index 588d069589..812ed92c64 100644
--- a/src/test/regress/expected/subselect.out
+++ b/src/test/regress/expected/subselect.out
@@ -1154,3 +1154,12 @@ fetch backward all in c1;
(2 rows)
commit;
+explain verbose
+with x as (select 1 as y)
+select * from (with x as (select 2 as y) select * from x) ss;
+ QUERY PLAN
+------------------------------------------
+ Result (cost=0.00..0.01 rows=1 width=4)
+ Output: 2
+(2 rows)
+
diff --git a/src/test/regress/sql/rowsecurity.sql b/src/test/regress/sql/rowsecurity.sql
index 38e9b38bc4..4c9e846fab 100644
--- a/src/test/regress/sql/rowsecurity.sql
+++ b/src/test/regress/sql/rowsecurity.sql
@@ -840,10 +840,10 @@ EXPLAIN (COSTS OFF) SELECT * FROM z1 WHERE f_leak(b);
PREPARE plancache_test AS SELECT * FROM z1 WHERE f_leak(b);
EXPLAIN (COSTS OFF) EXECUTE plancache_test;
-PREPARE plancache_test2 AS WITH q AS (SELECT * FROM z1 WHERE f_leak(b)) SELECT * FROM q,z2;
+PREPARE plancache_test2 AS WITH q AS MATERIALIZED (SELECT * FROM z1 WHERE f_leak(b)) SELECT * FROM q,z2;
EXPLAIN (COSTS OFF) EXECUTE plancache_test2;
-PREPARE plancache_test3 AS WITH q AS (SELECT * FROM z2) SELECT * FROM q,z1 WHERE f_leak(z1.b);
+PREPARE plancache_test3 AS WITH q AS MATERIALIZED (SELECT * FROM z2) SELECT * FROM q,z1 WHERE f_leak(z1.b);
EXPLAIN (COSTS OFF) EXECUTE plancache_test3;
SET ROLE regress_rls_group1;
@@ -1072,7 +1072,7 @@ INSERT INTO t1 (SELECT x, md5(x::text) FROM generate_series(0,20) x);
SET SESSION AUTHORIZATION regress_rls_bob;
WITH cte1 AS (SELECT * FROM t1 WHERE f_leak(b)) SELECT * FROM cte1;
-EXPLAIN (COSTS OFF) WITH cte1 AS (SELECT * FROM t1 WHERE f_leak(b)) SELECT * FROM cte1;
+EXPLAIN (COSTS OFF) WITH cte1 AS MATERIALIZED (SELECT * FROM t1 WHERE f_leak(b)) SELECT * FROM cte1;
WITH cte1 AS (UPDATE t1 SET a = a + 1 RETURNING *) SELECT * FROM cte1; --fail
WITH cte1 AS (UPDATE t1 SET a = a RETURNING *) SELECT * FROM cte1; --ok
diff --git a/src/test/regress/sql/rowtypes.sql b/src/test/regress/sql/rowtypes.sql
index e6d389805c..b3b07746c3 100644
--- a/src/test/regress/sql/rowtypes.sql
+++ b/src/test/regress/sql/rowtypes.sql
@@ -428,12 +428,12 @@ from (values (1,row(1,2)), (1,row(null,null)), (1,null),
(null,row(1,2)), (null,row(null,null)), (null,null) ) r(a,b);
explain (verbose, costs off)
-with r(a,b) as
+with r(a,b) as materialized
(values (1,row(1,2)), (1,row(null,null)), (1,null),
(null,row(1,2)), (null,row(null,null)), (null,null) )
select r, r is null as isnull, r is not null as isnotnull from r;
-with r(a,b) as
+with r(a,b) as materialized
(values (1,row(1,2)), (1,row(null,null)), (1,null),
(null,row(1,2)), (null,row(null,null)), (null,null) )
select r, r is null as isnull, r is not null as isnotnull from r;
diff --git a/src/test/regress/sql/rules.sql b/src/test/regress/sql/rules.sql
index f4ee30ec8f..82750b8e6f 100644
--- a/src/test/regress/sql/rules.sql
+++ b/src/test/regress/sql/rules.sql
@@ -1132,7 +1132,7 @@ SELECT tablename, rulename, definition FROM pg_rules
explain (costs off) INSERT INTO hats VALUES ('h8', 'forbidden') RETURNING *;
-- ensure upserting into a rule, with a CTE (different offsets!) works
-WITH data(hat_name, hat_color) AS (
+WITH data(hat_name, hat_color) AS MATERIALIZED (
VALUES ('h8', 'green'),
('h9', 'blue'),
('h7', 'forbidden')
@@ -1140,7 +1140,7 @@ WITH data(hat_name, hat_color) AS (
INSERT INTO hats
SELECT * FROM data
RETURNING *;
-EXPLAIN (costs off) WITH data(hat_name, hat_color) AS (
+EXPLAIN (costs off) WITH data(hat_name, hat_color) AS MATERIALIZED (
VALUES ('h8', 'green'),
('h9', 'blue'),
('h7', 'forbidden')
diff --git a/src/test/regress/sql/subselect.sql b/src/test/regress/sql/subselect.sql
index 843f511b3d..a1f9a7d890 100644
--- a/src/test/regress/sql/subselect.sql
+++ b/src/test/regress/sql/subselect.sql
@@ -625,3 +625,7 @@ move forward all in c1;
fetch backward all in c1;
commit;
+
+explain verbose
+with x as (select 1 as y)
+select * from (with x as (select 2 as y) select * from x) ss;
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
@ 2019-01-01 00:42 ` Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-02 04:31 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
1 sibling, 1 reply; 112+ messages in thread
From: Andrew Gierth @ 2019-01-01 00:42 UTC (permalink / raw)
To: Andreas Karlsson <andreas@proxel.se>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; David Fetter <david@fetter.org>; Thomas Munro <thomas.munro@enterprisedb.com>; pgsql-hackers
>>>>> "Andreas" == Andreas Karlsson <andreas@proxel.se> writes:
Andreas> I believe I have fixed these except for the comment on the
Andreas> conditions for when we inline.
Andreas> Andrew Gierth: Why did you chose to not inline on FOR UPDATE
Andreas> but inline volatile functions? I feel that this might be
Andreas> inconsistent since in both cases the query in the CTE can
Andreas> change behavior if the planner pushes a WHERE clause into the
Andreas> subquery, but maybe I am missing something.
I chose not to inline FOR UPDATE because it was an obvious compatibility
break, potentially changing the set of locked rows, and it was an easy
condition to test.
I did not test for volatile functions simply because this was a very
early stage of the project (which wasn't my project, I was just
assisting someone else). I left the comment "this likely needs some
additional checks" there for a reason.
--
Andrew (irc:RhodiumToad)
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 00:42 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
@ 2019-01-02 04:31 ` Andreas Karlsson <andreas@proxel.se>
0 siblings, 0 replies; 112+ messages in thread
From: Andreas Karlsson @ 2019-01-02 04:31 UTC (permalink / raw)
To: Andrew Gierth <andrew@tao11.riddles.org.uk>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; David Fetter <david@fetter.org>; Thomas Munro <thomas.munro@enterprisedb.com>; pgsql-hackers
On 1/1/19 1:42 AM, Andrew Gierth wrote:
>>>>>> "Andreas" == Andreas Karlsson <andreas@proxel.se> writes:
>
> Andreas> I believe I have fixed these except for the comment on the
> Andreas> conditions for when we inline.
>
> Andreas> Andrew Gierth: Why did you chose to not inline on FOR UPDATE
> Andreas> but inline volatile functions? I feel that this might be
> Andreas> inconsistent since in both cases the query in the CTE can
> Andreas> change behavior if the planner pushes a WHERE clause into the
> Andreas> subquery, but maybe I am missing something.
>
> I chose not to inline FOR UPDATE because it was an obvious compatibility
> break, potentially changing the set of locked rows, and it was an easy
> condition to test.
>
> I did not test for volatile functions simply because this was a very
> early stage of the project (which wasn't my project, I was just
> assisting someone else). I left the comment "this likely needs some
> additional checks" there for a reason.
Thanks, that makes sense! I will need to ponder some on if the behavior
change when predicates are pushed into a subquery with volatile
functions is ok. I am leaning towards no, because otherwise inlining
CTEs would affect more than query performance.
Andreas
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
@ 2019-01-01 02:18 ` Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-02 04:29 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
1 sibling, 2 replies; 112+ messages in thread
From: Andrew Gierth @ 2019-01-01 02:18 UTC (permalink / raw)
To: Andreas Karlsson <andreas@proxel.se>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; David Fetter <david@fetter.org>; Thomas Munro <thomas.munro@enterprisedb.com>; pgsql-hackers
>>>>> "Andreas" == Andreas Karlsson <andreas@proxel.se> writes:
Andreas> + if (rte->rtekind == RTE_CTE &&
Andreas> + strcmp(rte->ctename, context->ctename) == 0 &&
Andreas> + rte->ctelevelsup == context->levelsup)
Andreas> + {
Andreas> + Query *newquery = copyObject(context->ctequery);
Andreas> +
Andreas> + /* Preserve outer references, for example to other CTEs */
Andreas> + if (context->levelsup > 0)
Andreas> + IncrementVarSublevelsUp((Node *) newquery, context->levelsup, 1);
I had a comment around here which seems to have been lost:
* Secondly, views (and explicit subqueries) currently have
* different behaviour w.r.t. SELECT FOR UPDATE than CTEs do. A
* FOR UPDATE clause is treated as extending into views and
* subqueries, but not into CTEs. We preserve this distinction
* by not trying to push rowmarks into the new subquery.
This comment seems to me to be worth preserving (unless this behavior is
changed). What I'm referring to is the following, which is unchanged by
the patch:
create table t1 as select 123 as a;
create view v1 as select * from t1;
select * from t1 for update; -- locks row in t1
select * from t1 for update of t1; -- locks row in t1
select * from v1 for update; -- locks row in t1
select * from v1 for update of v1; -- locks row in t1
select * from (select * from t1) s1 for update; -- locks row in t1
select * from (select * from t1) s1 for update of s1; -- locks row in t1
with c1 as (select * from t1)
select * from c1 for update; -- does NOT lock anything at all
with c1 as (select * from t1)
select * from c1 for update of c1; -- parse-time error
(Obviously, inlining decisions should not change what gets locked;
the behavior here should not be changed unless it is changed for both
inlined and non-inlined CTEs.)
--
Andrew (irc:RhodiumToad)
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
@ 2019-01-02 04:29 ` Andreas Karlsson <andreas@proxel.se>
1 sibling, 0 replies; 112+ messages in thread
From: Andreas Karlsson @ 2019-01-02 04:29 UTC (permalink / raw)
To: Andrew Gierth <andrew@tao11.riddles.org.uk>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; David Fetter <david@fetter.org>; Thomas Munro <thomas.munro@enterprisedb.com>; pgsql-hackers
On 1/1/19 3:18 AM, Andrew Gierth wrote:
> I had a comment around here which seems to have been lost:
>
> * Secondly, views (and explicit subqueries) currently have
> * different behaviour w.r.t. SELECT FOR UPDATE than CTEs do. A
> * FOR UPDATE clause is treated as extending into views and
> * subqueries, but not into CTEs. We preserve this distinction
> * by not trying to push rowmarks into the new subquery.
>
> This comment seems to me to be worth preserving (unless this behavior is
> changed). What I'm referring to is the following, which is unchanged by
> the patch:
>
> create table t1 as select 123 as a;
> create view v1 as select * from t1;
> select * from t1 for update; -- locks row in t1
> select * from t1 for update of t1; -- locks row in t1
> select * from v1 for update; -- locks row in t1
> select * from v1 for update of v1; -- locks row in t1
> select * from (select * from t1) s1 for update; -- locks row in t1
> select * from (select * from t1) s1 for update of s1; -- locks row in t1
> with c1 as (select * from t1)
> select * from c1 for update; -- does NOT lock anything at all
> with c1 as (select * from t1)
> select * from c1 for update of c1; -- parse-time error
>
> (Obviously, inlining decisions should not change what gets locked;
> the behavior here should not be changed unless it is changed for both
> inlined and non-inlined CTEs.)
I see, I misread the comment. I will re-add it, possibly with some word
smithing. Thanks!
Andreas
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
@ 2019-01-10 01:28 ` Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-22 03:47 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
1 sibling, 2 replies; 112+ messages in thread
From: Andreas Karlsson @ 2019-01-10 01:28 UTC (permalink / raw)
To: Andrew Gierth <andrew@tao11.riddles.org.uk>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; David Fetter <david@fetter.org>; Thomas Munro <thomas.munro@enterprisedb.com>; pgsql-hackers
Here is a new version of the patch with added tests, improved comments,
some minor code cleanup and most importantly slightly changed logic for
when we should inline.
The current strategy is to always inline unless the CTE is recursive or
it has side effects, i.e. it is a DML query, it contains calls it a
volatile function, or it contains FOR SHARE/FOR UPDATE. I feel this is a
conservative approach which prevents behavioral changes (other than
performance).
Current open issues:
1. Currently the patch will inline CTEs even when there are multiple
references to them. If this is a win or not depends on the specific
query so I am not sure what we should do here. One thing worth noting is
that our current documentation talks a lot about how CTEs are only
evaluated once.
"A useful property of <literal>WITH</literal> queries is that they are
evaluated
only once per execution of the parent query, even if they are referred to
more than once by the parent query or sibling <literal>WITH</literal>
queries.
Thus, expensive calculations that are needed in multiple places can be
placed within a <literal>WITH</literal> query to avoid redundant work.
Another
possible application is to prevent unwanted multiple evaluations of
functions with side-effects."
What do you think?
2. Feedback on the new syntax. I am personally fine with the current
syntax, but it was just something I just quickly hacked together to move
the patch forward and which also solved my personal uses cases.
3. Are we happy with how I modified query_tree_walker()? I feel the code
would be clearer if we could change the tree walker to treat the RTE as
the parent node of the subquery instead of a sibling, but this seems
like potentially a quite invasive change.
4. I need to update the user documentation.
Andreas
Attachments:
[text/x-patch] inlining-ctes-v7.patch (29.2K, ../../d0619160-10af-dcc9-4e7d-57845b92570c@proxel.se/2-inlining-ctes-v7.patch)
download | inline diff:
commit 60aab4af243cc93d504b99c99010593c02a5705c
Author: Andreas Karlsson <andreas@proxel.se>
Date: Mon Sep 17 03:42:40 2018 +0200
CTE inlining
diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out
index bb92d9d37a..8c26dd1f26 100644
--- a/contrib/postgres_fdw/expected/postgres_fdw.out
+++ b/contrib/postgres_fdw/expected/postgres_fdw.out
@@ -1888,7 +1888,7 @@ SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) ORDER BY t1.c3, t
-- join in CTE
EXPLAIN (VERBOSE, COSTS OFF)
-WITH t (c1_1, c1_3, c2_1) AS (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
+WITH t (c1_1, c1_3, c2_1) AS MATERIALIZED (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
QUERY PLAN
-------------------------------------------------------------------------------------------------------------------------------------
Limit
@@ -1905,7 +1905,7 @@ WITH t (c1_1, c1_3, c2_1) AS (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2
Output: t.c1_1, t.c2_1, t.c1_3
(12 rows)
-WITH t (c1_1, c1_3, c2_1) AS (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
+WITH t (c1_1, c1_3, c2_1) AS MATERIALIZED (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
c1_1 | c2_1
------+------
101 | 101
diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql
index f438165650..56602a164c 100644
--- a/contrib/postgres_fdw/sql/postgres_fdw.sql
+++ b/contrib/postgres_fdw/sql/postgres_fdw.sql
@@ -493,8 +493,8 @@ SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) ORDER BY t1.c3, t
SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) ORDER BY t1.c3, t1.c1 OFFSET 100 LIMIT 10 FOR SHARE;
-- join in CTE
EXPLAIN (VERBOSE, COSTS OFF)
-WITH t (c1_1, c1_3, c2_1) AS (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
-WITH t (c1_1, c1_3, c2_1) AS (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
+WITH t (c1_1, c1_3, c2_1) AS MATERIALIZED (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
+WITH t (c1_1, c1_3, c2_1) AS MATERIALIZED (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
-- ctid with whole-row reference
EXPLAIN (VERBOSE, COSTS OFF)
SELECT t1.ctid, t1, t2, t1.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) ORDER BY t1.c3, t1.c1 OFFSET 100 LIMIT 10;
diff --git a/doc/src/sgml/queries.sgml b/doc/src/sgml/queries.sgml
index 88bc189646..92f180c5cf 100644
--- a/doc/src/sgml/queries.sgml
+++ b/doc/src/sgml/queries.sgml
@@ -2199,6 +2199,8 @@ SELECT n FROM t LIMIT 100;
</para>
<para>
+ TODO: Update this for inlining and <literal>MATERIALIZED</literal>.
+
A useful property of <literal>WITH</literal> queries is that they are evaluated
only once per execution of the parent query, even if they are referred to
more than once by the parent query or sibling <literal>WITH</literal> queries.
diff --git a/doc/src/sgml/ref/select.sgml b/doc/src/sgml/ref/select.sgml
index 4db8142afa..92ede4fc6c 100644
--- a/doc/src/sgml/ref/select.sgml
+++ b/doc/src/sgml/ref/select.sgml
@@ -72,7 +72,7 @@ SELECT [ ALL | DISTINCT [ ON ( <replaceable class="parameter">expression</replac
<phrase>and <replaceable class="parameter">with_query</replaceable> is:</phrase>
- <replaceable class="parameter">with_query_name</replaceable> [ ( <replaceable class="parameter">column_name</replaceable> [, ...] ) ] AS ( <replaceable class="parameter">select</replaceable> | <replaceable class="parameter">values</replaceable> | <replaceable class="parameter">insert</replaceable> | <replaceable class="parameter">update</replaceable> | <replaceable class="parameter">delete</replaceable> )
+ <replaceable class="parameter">with_query_name</replaceable> [ ( <replaceable class="parameter">column_name</replaceable> [, ...] ) ] AS [ MATERIALIZED ] ( <replaceable class="parameter">select</replaceable> | <replaceable class="parameter">values</replaceable> | <replaceable class="parameter">insert</replaceable> | <replaceable class="parameter">update</replaceable> | <replaceable class="parameter">delete</replaceable> )
TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ]
</synopsis>
@@ -273,6 +273,8 @@ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ]
</para>
<para>
+ TODO: Update this for inlining and <literal>MATERIALIZED</literal>.
+
A key property of <literal>WITH</literal> queries is that they
are evaluated only once per execution of the primary query,
even if the primary query refers to them more than once.
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 006a3d1772..19b21ab6c5 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2539,6 +2539,7 @@ _copyCommonTableExpr(const CommonTableExpr *from)
COPY_NODE_FIELD(ctequery);
COPY_LOCATION_FIELD(location);
COPY_SCALAR_FIELD(cterecursive);
+ COPY_SCALAR_FIELD(ctematerialized);
COPY_SCALAR_FIELD(cterefcount);
COPY_NODE_FIELD(ctecolnames);
COPY_NODE_FIELD(ctecoltypes);
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index 133df1b364..6779adfe8d 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2793,6 +2793,7 @@ _equalCommonTableExpr(const CommonTableExpr *a, const CommonTableExpr *b)
COMPARE_NODE_FIELD(ctequery);
COMPARE_LOCATION_FIELD(location);
COMPARE_SCALAR_FIELD(cterecursive);
+ COMPARE_SCALAR_FIELD(ctematerialized);
COMPARE_SCALAR_FIELD(cterefcount);
COMPARE_NODE_FIELD(ctecolnames);
COMPARE_NODE_FIELD(ctecoltypes);
diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c
index 19b65f681d..03d085eda4 100644
--- a/src/backend/nodes/nodeFuncs.c
+++ b/src/backend/nodes/nodeFuncs.c
@@ -2314,8 +2314,7 @@ range_table_walker(List *rtable,
{
RangeTblEntry *rte = (RangeTblEntry *) lfirst(rt);
- /* For historical reasons, visiting RTEs is not the default */
- if (flags & QTW_EXAMINE_RTES)
+ if (flags & QTW_EXAMINE_RTES_BEFORE)
if (walker(rte, context))
return true;
@@ -2353,6 +2352,10 @@ range_table_walker(List *rtable,
break;
}
+ if (flags & QTW_EXAMINE_RTES_AFTER)
+ if (walker(rte, context))
+ return true;
+
if (walker(rte->securityQuals, context))
return true;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 0fde876c77..764bf36674 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2983,6 +2983,7 @@ _outCommonTableExpr(StringInfo str, const CommonTableExpr *node)
WRITE_NODE_FIELD(ctequery);
WRITE_LOCATION_FIELD(location);
WRITE_BOOL_FIELD(cterecursive);
+ WRITE_BOOL_FIELD(ctematerialized);
WRITE_INT_FIELD(cterefcount);
WRITE_NODE_FIELD(ctecolnames);
WRITE_NODE_FIELD(ctecoltypes);
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index ec6f2569ab..bd9086cb3f 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -421,6 +421,7 @@ _readCommonTableExpr(void)
READ_NODE_FIELD(ctequery);
READ_LOCATION_FIELD(location);
READ_BOOL_FIELD(cterecursive);
+ READ_BOOL_FIELD(ctematerialized);
READ_INT_FIELD(cterefcount);
READ_NODE_FIELD(ctecolnames);
READ_NODE_FIELD(ctecoltypes);
diff --git a/src/backend/optimizer/plan/setrefs.c b/src/backend/optimizer/plan/setrefs.c
index fc7ba0dea2..a31c287c35 100644
--- a/src/backend/optimizer/plan/setrefs.c
+++ b/src/backend/optimizer/plan/setrefs.c
@@ -340,7 +340,7 @@ flatten_unplanned_rtes(PlannerGlobal *glob, RangeTblEntry *rte)
(void) query_tree_walker(rte->subquery,
flatten_rtes_walker,
(void *) glob,
- QTW_EXAMINE_RTES);
+ QTW_EXAMINE_RTES_BEFORE);
}
static bool
@@ -363,7 +363,7 @@ flatten_rtes_walker(Node *node, PlannerGlobal *glob)
return query_tree_walker((Query *) node,
flatten_rtes_walker,
(void *) glob,
- QTW_EXAMINE_RTES);
+ QTW_EXAMINE_RTES_BEFORE);
}
return expression_tree_walker(node, flatten_rtes_walker,
(void *) glob);
diff --git a/src/backend/optimizer/plan/subselect.c b/src/backend/optimizer/plan/subselect.c
index cb1ea86afb..019dce7f67 100644
--- a/src/backend/optimizer/plan/subselect.c
+++ b/src/backend/optimizer/plan/subselect.c
@@ -53,6 +53,14 @@ typedef struct finalize_primnode_context
Bitmapset *paramids; /* Non-local PARAM_EXEC paramids found */
} finalize_primnode_context;
+typedef struct inline_cte_walker_context
+{
+ const char *ctename;
+ int levelsup;
+ int refcount;
+ Query *ctequery;
+ CommonTableExpr *cte;
+} inline_cte_walker_context;
static Node *build_subplan(PlannerInfo *root, Plan *plan, PlannerInfo *subroot,
List *plan_params,
@@ -1145,14 +1153,129 @@ hash_ok_operator(OpExpr *expr)
}
}
+/*
+ * inline_cte_walker: recursively walk the query and inline CTE
+ *
+ * While walking the expression tree with take care to examine the RTEs and
+ * potentially replace a CTE RTE with a subquery RTE after recursing into the
+ * RTE to avoid recursing into the newly inlined CTE.
+ */
+static bool
+inline_cte_walker(Node *node, inline_cte_walker_context *context)
+{
+ if (!node)
+ return false;
+
+ if (IsA(node, Query))
+ {
+ Query *query = castNode(Query, node);
+
+ context->levelsup++;
+
+ /*
+ * Visit the actual RTE nodes after their contents so we can modify
+ * the RTE node without having to visit the newly inlined contents.
+ */
+ query_tree_walker(query, inline_cte_walker, context,
+ QTW_EXAMINE_RTES_AFTER);
+
+ context->levelsup--;
+
+ return false;
+ }
+ else if (IsA(node, RangeTblEntry))
+ {
+ RangeTblEntry *rte = castNode(RangeTblEntry, node);
+
+ if (rte->rtekind == RTE_CTE &&
+ strcmp(rte->ctename, context->ctename) == 0 &&
+ rte->ctelevelsup == context->levelsup)
+ {
+ Query *newquery = copyObject(context->ctequery);
+
+ /* Preserve outer references, for example to other CTEs */
+ if (context->levelsup > 0)
+ IncrementVarSublevelsUp((Node *) newquery, context->levelsup, 1);
+
+ /*
+ * Replace the CTE reference in the range table with a subquery
+ *
+ * A FOR UPDATE clause is treated as extending into views and
+ * subqueries, but not into CTEs. We preserve this distinction
+ * by not trying to push rowmarks into the new subquery.
+ */
+ rte->rtekind = RTE_SUBQUERY;
+ rte->subquery = newquery;
+ rte->security_barrier = false;
+
+ /* Zero out CTE specific fields */
+ rte->ctename = NULL;
+ rte->ctelevelsup = 0;
+ rte->self_reference = false;
+ rte->coltypes = NIL;
+ rte->coltypmods = NIL;
+ rte->colcollations = NIL;
+
+ context->refcount--;
+ }
+
+ return false;
+ }
+
+ return expression_tree_walker(node, inline_cte_walker, context);
+}
+
+/*
+ * inline_cte: replace all refernces to CTE with subqueries
+ */
+static void
+inline_cte(PlannerInfo *root, CommonTableExpr *cte)
+{
+ struct inline_cte_walker_context context;
+ context.ctequery = castNode(Query, cte->ctequery);
+ context.ctename = cte->ctename;
+ context.refcount = cte->cterefcount;
+ context.levelsup = -1;
+ context.cte = cte;
+ inline_cte_walker((Node *) root->parse, &context);
+ /* we must replace all references */
+ Assert(context.refcount == 0);
+}
+
+static bool
+contain_row_marks_walker(Node *node, void *context)
+{
+ if (!node)
+ return false;
+
+ if (IsA(node, Query))
+ {
+ Query *query = castNode(Query, node);
+
+ if (query->rowMarks != NIL)
+ return true;
+
+ return query_tree_walker(query, contain_row_marks_walker, context, 0);
+ }
+ return expression_tree_walker(node, contain_row_marks_walker, context);
+}
+
+/*
+ * contain_row_marks: check the query or any subquery contains row marks
+ */
+static bool
+contain_row_marks(Node *node)
+{
+ return contain_row_marks_walker(node, NULL);
+}
/*
* SS_process_ctes: process a query's WITH list
*
- * We plan each interesting WITH item and convert it to an initplan.
- * A side effect is to fill in root->cte_plan_ids with a list that
- * parallels root->parse->cteList and provides the subplan ID for
- * each CTE's initplan.
+ * We plan each interesting WITH item and either convert it to an initplan, or
+ * if allowed inline it as a subquery. A side effect is to fill in
+ * root->cte_plan_ids with a list that parallels root->parse->cteList and
+ * provides the subplan ID for each CTE's initplan, or a dummy ID when inlined.
*/
void
SS_process_ctes(PlannerInfo *root)
@@ -1184,6 +1307,27 @@ SS_process_ctes(PlannerInfo *root)
}
/*
+ * We always inline the CTE as a subquery as long as it has no side
+ * effects, whch in this context means that it is a SELECT which does
+ * not call any volatile functions and does not have any rowmarks.
+ *
+ * The user may also have requested to not inline the CTE with the
+ * MATERIALIZED keyword.
+ */
+ if (!cte->ctematerialized &&
+ cmdType == CMD_SELECT &&
+ !cte->cterecursive &&
+ !contain_volatile_functions(cte->ctequery) &&
+ !contain_row_marks(cte->ctequery))
+ {
+ inline_cte(root, cte);
+
+ /* Make a dummy entry in cte_plan_ids */
+ root->cte_plan_ids = lappend_int(root->cte_plan_ids, -1);
+ continue;
+ }
+
+ /*
* Copy the source Query node. Probably not necessary, but let's keep
* this similar to make_subplan.
*/
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index c086235b25..623f582a57 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -518,7 +518,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
Bit ConstBit BitWithLength BitWithoutLength
%type <str> character
%type <str> extract_arg
-%type <boolean> opt_varying opt_timezone opt_no_inherit
+%type <boolean> opt_varying opt_timezone opt_no_inherit opt_materialized
%type <ival> Iconst SignedIconst
%type <str> Sconst comment_text notify_payload
@@ -11368,17 +11368,23 @@ cte_list:
| cte_list ',' common_table_expr { $$ = lappend($1, $3); }
;
-common_table_expr: name opt_name_list AS '(' PreparableStmt ')'
+common_table_expr: name opt_name_list AS opt_materialized '(' PreparableStmt ')'
{
CommonTableExpr *n = makeNode(CommonTableExpr);
n->ctename = $1;
n->aliascolnames = $2;
- n->ctequery = $5;
+ n->ctequery = $6;
+ n->ctematerialized = $4;
n->location = @1;
$$ = (Node *) n;
}
;
+opt_materialized:
+ MATERIALIZED { $$ = true; }
+ | /*EMPTY*/ { $$ = false; }
+ ;
+
opt_with_clause:
with_clause { $$ = $1; }
| /*EMPTY*/ { $$ = NULL; }
diff --git a/src/backend/rewrite/rewriteManip.c b/src/backend/rewrite/rewriteManip.c
index acbe669294..fd7a751c9a 100644
--- a/src/backend/rewrite/rewriteManip.c
+++ b/src/backend/rewrite/rewriteManip.c
@@ -761,7 +761,7 @@ IncrementVarSublevelsUp_walker(Node *node,
result = query_tree_walker((Query *) node,
IncrementVarSublevelsUp_walker,
(void *) context,
- QTW_EXAMINE_RTES);
+ QTW_EXAMINE_RTES_BEFORE);
context->min_sublevels_up--;
return result;
}
@@ -785,7 +785,7 @@ IncrementVarSublevelsUp(Node *node, int delta_sublevels_up,
query_or_expression_tree_walker(node,
IncrementVarSublevelsUp_walker,
(void *) &context,
- QTW_EXAMINE_RTES);
+ QTW_EXAMINE_RTES_BEFORE);
}
/*
@@ -804,7 +804,7 @@ IncrementVarSublevelsUp_rtable(List *rtable, int delta_sublevels_up,
range_table_walker(rtable,
IncrementVarSublevelsUp_walker,
(void *) &context,
- QTW_EXAMINE_RTES);
+ QTW_EXAMINE_RTES_BEFORE);
}
diff --git a/src/backend/utils/adt/ruleutils.c b/src/backend/utils/adt/ruleutils.c
index 77811f6818..5b4c21e4f9 100644
--- a/src/backend/utils/adt/ruleutils.c
+++ b/src/backend/utils/adt/ruleutils.c
@@ -5165,7 +5165,10 @@ get_with_clause(Query *query, deparse_context *context)
}
appendStringInfoChar(buf, ')');
}
- appendStringInfoString(buf, " AS (");
+ appendStringInfoString(buf, " AS ");
+ if (cte->ctematerialized)
+ appendStringInfoString(buf, "MATERIALIZED ");
+ appendStringInfoChar(buf, '(');
if (PRETTY_INDENT(context))
appendContextKeyword(context, "", 0, 0, 0);
get_query_def((Query *) cte->ctequery, buf, context->namespaces, NULL,
diff --git a/src/include/nodes/nodeFuncs.h b/src/include/nodes/nodeFuncs.h
index 7739600db2..2774f15cc8 100644
--- a/src/include/nodes/nodeFuncs.h
+++ b/src/include/nodes/nodeFuncs.h
@@ -22,8 +22,10 @@
#define QTW_IGNORE_RC_SUBQUERIES 0x03 /* both of above */
#define QTW_IGNORE_JOINALIASES 0x04 /* JOIN alias var lists */
#define QTW_IGNORE_RANGE_TABLE 0x08 /* skip rangetable entirely */
-#define QTW_EXAMINE_RTES 0x10 /* examine RTEs */
+#define QTW_EXAMINE_RTES_BEFORE 0x10 /* examine RTE nodes before their contents */
#define QTW_DONT_COPY_QUERY 0x20 /* do not copy top Query */
+#define QTW_EXAMINE_RTES_AFTER 0x40 /* examine RTE nodes after their contents */
+
/* callback function for check_functions_in_node */
typedef bool (*check_function_callback) (Oid func_id, void *context);
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 27782fed6c..42f226e459 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -1409,6 +1409,7 @@ typedef struct CommonTableExpr
int location; /* token location, or -1 if unknown */
/* These fields are set during parse analysis: */
bool cterecursive; /* is this CTE actually recursive? */
+ bool ctematerialized; /* is this an optimization fence? */
int cterefcount; /* number of RTEs referencing this CTE
* (excluding internal self-references) */
List *ctecolnames; /* list of output column names */
diff --git a/src/test/regress/expected/rowsecurity.out b/src/test/regress/expected/rowsecurity.out
index 1d12b01068..1a0932d863 100644
--- a/src/test/regress/expected/rowsecurity.out
+++ b/src/test/regress/expected/rowsecurity.out
@@ -2179,7 +2179,7 @@ EXPLAIN (COSTS OFF) EXECUTE plancache_test;
Filter: (((a % 2) = 0) AND f_leak(b))
(2 rows)
-PREPARE plancache_test2 AS WITH q AS (SELECT * FROM z1 WHERE f_leak(b)) SELECT * FROM q,z2;
+PREPARE plancache_test2 AS WITH q AS MATERIALIZED (SELECT * FROM z1 WHERE f_leak(b)) SELECT * FROM q,z2;
EXPLAIN (COSTS OFF) EXECUTE plancache_test2;
QUERY PLAN
-------------------------------------------------
@@ -2192,7 +2192,7 @@ EXPLAIN (COSTS OFF) EXECUTE plancache_test2;
-> Seq Scan on z2
(7 rows)
-PREPARE plancache_test3 AS WITH q AS (SELECT * FROM z2) SELECT * FROM q,z1 WHERE f_leak(z1.b);
+PREPARE plancache_test3 AS WITH q AS MATERIALIZED (SELECT * FROM z2) SELECT * FROM q,z1 WHERE f_leak(z1.b);
EXPLAIN (COSTS OFF) EXECUTE plancache_test3;
QUERY PLAN
-----------------------------------------------------
@@ -2853,7 +2853,7 @@ NOTICE: f_leak => 98f13708210194c475687be6106a3b84
20 | 98f13708210194c475687be6106a3b84
(11 rows)
-EXPLAIN (COSTS OFF) WITH cte1 AS (SELECT * FROM t1 WHERE f_leak(b)) SELECT * FROM cte1;
+EXPLAIN (COSTS OFF) WITH cte1 AS MATERIALIZED (SELECT * FROM t1 WHERE f_leak(b)) SELECT * FROM cte1;
QUERY PLAN
-------------------------------------------------
CTE Scan on cte1
diff --git a/src/test/regress/expected/rowtypes.out b/src/test/regress/expected/rowtypes.out
index d6a1a3331e..14757b40f6 100644
--- a/src/test/regress/expected/rowtypes.out
+++ b/src/test/regress/expected/rowtypes.out
@@ -1037,7 +1037,7 @@ from (values (1,row(1,2)), (1,row(null,null)), (1,null),
(6 rows)
explain (verbose, costs off)
-with r(a,b) as
+with r(a,b) as materialized
(values (1,row(1,2)), (1,row(null,null)), (1,null),
(null,row(1,2)), (null,row(null,null)), (null,null) )
select r, r is null as isnull, r is not null as isnotnull from r;
@@ -1050,7 +1050,7 @@ select r, r is null as isnull, r is not null as isnotnull from r;
Output: "*VALUES*".column1, "*VALUES*".column2
(5 rows)
-with r(a,b) as
+with r(a,b) as materialized
(values (1,row(1,2)), (1,row(null,null)), (1,null),
(null,row(1,2)), (null,row(null,null)), (null,null) )
select r, r is null as isnull, r is not null as isnotnull from r;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index e384cd2279..32dfab94b2 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -3108,7 +3108,7 @@ explain (costs off) INSERT INTO hats VALUES ('h8', 'forbidden') RETURNING *;
(5 rows)
-- ensure upserting into a rule, with a CTE (different offsets!) works
-WITH data(hat_name, hat_color) AS (
+WITH data(hat_name, hat_color) AS MATERIALIZED (
VALUES ('h8', 'green'),
('h9', 'blue'),
('h7', 'forbidden')
@@ -3122,7 +3122,7 @@ RETURNING *;
h9 | blue
(2 rows)
-EXPLAIN (costs off) WITH data(hat_name, hat_color) AS (
+EXPLAIN (costs off) WITH data(hat_name, hat_color) AS MATERIALIZED (
VALUES ('h8', 'green'),
('h9', 'blue'),
('h7', 'forbidden')
diff --git a/src/test/regress/expected/subselect.out b/src/test/regress/expected/subselect.out
index 588d069589..d52dfe022a 100644
--- a/src/test/regress/expected/subselect.out
+++ b/src/test/regress/expected/subselect.out
@@ -1154,3 +1154,92 @@ fetch backward all in c1;
(2 rows)
commit;
+-- Ensure that we inline the currect CTE when there multiple CTEs with the same name
+explain (verbose, costs off)
+with x as (select 1 as y)
+select * from (with x as (select 2 as y) select * from x) ss;
+ QUERY PLAN
+-------------
+ Result
+ Output: 2
+(2 rows)
+
+-- Ensure that we do not inline when there are side effects
+-- Basic subquery which can be inlined
+explain (verbose, costs off)
+with x as (select * from (select f1 from subselect_tbl) ss)
+select * from x where f1 = 1;
+ QUERY PLAN
+----------------------------------
+ Seq Scan on public.subselect_tbl
+ Output: subselect_tbl.f1
+ Filter: (subselect_tbl.f1 = 1)
+(3 rows)
+
+-- Stable functions are safe to inline
+explain (verbose, costs off)
+with x as (select * from (select f1, now() from subselect_tbl) ss)
+select * from x where f1 = 1;
+ QUERY PLAN
+-----------------------------------
+ Seq Scan on public.subselect_tbl
+ Output: subselect_tbl.f1, now()
+ Filter: (subselect_tbl.f1 = 1)
+(3 rows)
+
+-- Volatile functions prevent inlining
+explain (verbose, costs off)
+with x as (select * from (select f1, random() from subselect_tbl) ss)
+select * from x where f1 = 1;
+ QUERY PLAN
+----------------------------------------------
+ CTE Scan on x
+ Output: x.f1, x.random
+ Filter: (x.f1 = 1)
+ CTE x
+ -> Seq Scan on public.subselect_tbl
+ Output: subselect_tbl.f1, random()
+(6 rows)
+
+-- Row marks prevent inlining
+explain (verbose, costs off)
+with x as (select * from (select f1 from subselect_tbl for update) ss)
+select * from x where f1 = 1;
+ QUERY PLAN
+--------------------------------------------------------------------
+ CTE Scan on x
+ Output: x.f1
+ Filter: (x.f1 = 1)
+ CTE x
+ -> Subquery Scan on ss
+ Output: ss.f1
+ -> LockRows
+ Output: subselect_tbl.f1, subselect_tbl.ctid
+ -> Seq Scan on public.subselect_tbl
+ Output: subselect_tbl.f1, subselect_tbl.ctid
+(10 rows)
+
+-- Explicitly request materialization
+explain (verbose, costs off)
+with x as materialized (select * from (select f1 from subselect_tbl) ss)
+select * from x where f1 = 1;
+ QUERY PLAN
+------------------------------------------
+ CTE Scan on x
+ Output: x.f1
+ Filter: (x.f1 = 1)
+ CTE x
+ -> Seq Scan on public.subselect_tbl
+ Output: subselect_tbl.f1
+(6 rows)
+
+-- Row marks are not pushed into CTEs
+explain (verbose, costs off)
+with x as (select * from subselect_tbl)
+select * from x for update;
+ QUERY PLAN
+----------------------------------------------------------------
+ Seq Scan on public.subselect_tbl
+ Output: subselect_tbl.f1, subselect_tbl.f2, subselect_tbl.f3
+(2 rows)
+
diff --git a/src/test/regress/sql/rowsecurity.sql b/src/test/regress/sql/rowsecurity.sql
index 38e9b38bc4..4c9e846fab 100644
--- a/src/test/regress/sql/rowsecurity.sql
+++ b/src/test/regress/sql/rowsecurity.sql
@@ -840,10 +840,10 @@ EXPLAIN (COSTS OFF) SELECT * FROM z1 WHERE f_leak(b);
PREPARE plancache_test AS SELECT * FROM z1 WHERE f_leak(b);
EXPLAIN (COSTS OFF) EXECUTE plancache_test;
-PREPARE plancache_test2 AS WITH q AS (SELECT * FROM z1 WHERE f_leak(b)) SELECT * FROM q,z2;
+PREPARE plancache_test2 AS WITH q AS MATERIALIZED (SELECT * FROM z1 WHERE f_leak(b)) SELECT * FROM q,z2;
EXPLAIN (COSTS OFF) EXECUTE plancache_test2;
-PREPARE plancache_test3 AS WITH q AS (SELECT * FROM z2) SELECT * FROM q,z1 WHERE f_leak(z1.b);
+PREPARE plancache_test3 AS WITH q AS MATERIALIZED (SELECT * FROM z2) SELECT * FROM q,z1 WHERE f_leak(z1.b);
EXPLAIN (COSTS OFF) EXECUTE plancache_test3;
SET ROLE regress_rls_group1;
@@ -1072,7 +1072,7 @@ INSERT INTO t1 (SELECT x, md5(x::text) FROM generate_series(0,20) x);
SET SESSION AUTHORIZATION regress_rls_bob;
WITH cte1 AS (SELECT * FROM t1 WHERE f_leak(b)) SELECT * FROM cte1;
-EXPLAIN (COSTS OFF) WITH cte1 AS (SELECT * FROM t1 WHERE f_leak(b)) SELECT * FROM cte1;
+EXPLAIN (COSTS OFF) WITH cte1 AS MATERIALIZED (SELECT * FROM t1 WHERE f_leak(b)) SELECT * FROM cte1;
WITH cte1 AS (UPDATE t1 SET a = a + 1 RETURNING *) SELECT * FROM cte1; --fail
WITH cte1 AS (UPDATE t1 SET a = a RETURNING *) SELECT * FROM cte1; --ok
diff --git a/src/test/regress/sql/rowtypes.sql b/src/test/regress/sql/rowtypes.sql
index e6d389805c..b3b07746c3 100644
--- a/src/test/regress/sql/rowtypes.sql
+++ b/src/test/regress/sql/rowtypes.sql
@@ -428,12 +428,12 @@ from (values (1,row(1,2)), (1,row(null,null)), (1,null),
(null,row(1,2)), (null,row(null,null)), (null,null) ) r(a,b);
explain (verbose, costs off)
-with r(a,b) as
+with r(a,b) as materialized
(values (1,row(1,2)), (1,row(null,null)), (1,null),
(null,row(1,2)), (null,row(null,null)), (null,null) )
select r, r is null as isnull, r is not null as isnotnull from r;
-with r(a,b) as
+with r(a,b) as materialized
(values (1,row(1,2)), (1,row(null,null)), (1,null),
(null,row(1,2)), (null,row(null,null)), (null,null) )
select r, r is null as isnull, r is not null as isnotnull from r;
diff --git a/src/test/regress/sql/rules.sql b/src/test/regress/sql/rules.sql
index f4ee30ec8f..82750b8e6f 100644
--- a/src/test/regress/sql/rules.sql
+++ b/src/test/regress/sql/rules.sql
@@ -1132,7 +1132,7 @@ SELECT tablename, rulename, definition FROM pg_rules
explain (costs off) INSERT INTO hats VALUES ('h8', 'forbidden') RETURNING *;
-- ensure upserting into a rule, with a CTE (different offsets!) works
-WITH data(hat_name, hat_color) AS (
+WITH data(hat_name, hat_color) AS MATERIALIZED (
VALUES ('h8', 'green'),
('h9', 'blue'),
('h7', 'forbidden')
@@ -1140,7 +1140,7 @@ WITH data(hat_name, hat_color) AS (
INSERT INTO hats
SELECT * FROM data
RETURNING *;
-EXPLAIN (costs off) WITH data(hat_name, hat_color) AS (
+EXPLAIN (costs off) WITH data(hat_name, hat_color) AS MATERIALIZED (
VALUES ('h8', 'green'),
('h9', 'blue'),
('h7', 'forbidden')
diff --git a/src/test/regress/sql/subselect.sql b/src/test/regress/sql/subselect.sql
index 843f511b3d..bdbba3838f 100644
--- a/src/test/regress/sql/subselect.sql
+++ b/src/test/regress/sql/subselect.sql
@@ -625,3 +625,42 @@ move forward all in c1;
fetch backward all in c1;
commit;
+
+-- Ensure that we inline the currect CTE when there multiple CTEs with the same name
+
+explain (verbose, costs off)
+with x as (select 1 as y)
+select * from (with x as (select 2 as y) select * from x) ss;
+
+-- Ensure that we do not inline when there are side effects
+
+-- Basic subquery which can be inlined
+explain (verbose, costs off)
+with x as (select * from (select f1 from subselect_tbl) ss)
+select * from x where f1 = 1;
+
+-- Stable functions are safe to inline
+explain (verbose, costs off)
+with x as (select * from (select f1, now() from subselect_tbl) ss)
+select * from x where f1 = 1;
+
+-- Volatile functions prevent inlining
+explain (verbose, costs off)
+with x as (select * from (select f1, random() from subselect_tbl) ss)
+select * from x where f1 = 1;
+
+-- Row marks prevent inlining
+explain (verbose, costs off)
+with x as (select * from (select f1 from subselect_tbl for update) ss)
+select * from x where f1 = 1;
+
+-- Explicitly request materialization
+explain (verbose, costs off)
+with x as materialized (select * from (select f1 from subselect_tbl) ss)
+select * from x where f1 = 1;
+
+-- Row marks are not pushed into CTEs
+
+explain (verbose, costs off)
+with x as (select * from subselect_tbl)
+select * from x for update;
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
@ 2019-01-10 02:24 ` Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
1 sibling, 1 reply; 112+ messages in thread
From: Thomas Munro @ 2019-01-10 02:24 UTC (permalink / raw)
To: Andreas Karlsson <andreas@proxel.se>; +Cc: Andrew Gierth <andrew@tao11.riddles.org.uk>; Tom Lane <tgl@sss.pgh.pa.us>; David Fetter <david@fetter.org>; pgsql-hackers
On Thu, Jan 10, 2019 at 2:28 PM Andreas Karlsson <andreas@proxel.se> wrote:
> 2. Feedback on the new syntax. I am personally fine with the current
> syntax, but it was just something I just quickly hacked together to move
> the patch forward and which also solved my personal uses cases.
Thanks for working on this. I very much want to see this feature go
in. As mentioned by Andres up-thread, TPC-DS makes a lot of use of
CTEs... let me see, 34 queries out of 99 have a WITH clause. These
will hopefully become candidates for parallel query.
I know this is a thorny topic, but I have to say that I am uneasy
about the MATERIALIZED syntax. Here's how you write that in some
other RDBMS that loves hints:
WITH foo AS (SELECT /*+ MATERIALIZE */ ...)
I understood that it was a long standing project policy that we don't
want planner hints, but now we have a proposal to support one with a
top-level non-standard syntax. If we take this syntax, should we not
also accept MATERIALIZED in front of subselects?
-1
--
Thomas Munro
http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
@ 2019-01-11 16:12 ` Robert Haas <robertmhaas@gmail.com>
2019-01-11 18:19 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 18:49 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
0 siblings, 2 replies; 112+ messages in thread
From: Robert Haas @ 2019-01-11 16:12 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@enterprisedb.com>; +Cc: Andreas Karlsson <andreas@proxel.se>; Andrew Gierth <andrew@tao11.riddles.org.uk>; Tom Lane <tgl@sss.pgh.pa.us>; David Fetter <david@fetter.org>; pgsql-hackers
> I know this is a thorny topic, but I have to say that I am uneasy
> about the MATERIALIZED syntax. Here's how you write that in some
> other RDBMS that loves hints:
>
> WITH foo AS (SELECT /*+ MATERIALIZE */ ...)
>
> I understood that it was a long standing project policy that we don't
> want planner hints, but now we have a proposal to support one with a
> top-level non-standard syntax. If we take this syntax, should we not
> also accept MATERIALIZED in front of subselects?
>
> -1
I think this is unduly negative. Do you want to also remove the
IMMUTABLE, STABLE, and VOLATILE keywords from functions because those
are hints to the planner as to how those things should be treated?
Should we remove CREATE INDEX -- which is, after all, a non-standard
extension to SQL syntax -- because it presumes that the user is in a
better position than we are to know which columns ought to be indexed?
OK, I know that's a bit of a straw man -- you're talking about hints
within a query, not DDL. Still, I think our theory about not having
hints is that we should have the optimizer try to figure it out
instead of making the user specify the behavior that they want -- and
I think sometimes that's setting the bar at an impossible level. In
the case of things that have side effects, like FOR UPDATE/SHARE or
volatile functions, we really can't know the user's intention unless
the user tells us. But even in cases where there are no side effects
and it's just a question of finding the most efficient plan, it
doesn't seem crazy to me to allow the user to give us a clue about
what they have in mind.
It's not like this is a thing where we can get this right 90% of the
time and with some more planner work we can drive it up to near 100%.
We're going to be wrong a lot, even if we do expensive things like try
planning it both ways and see which one comes out cheaper on cost, and
we don't like driving up planner CPU consumption, either. So it seems
to me that letting the user say what they want is a very pragmatic
approach. Sometimes, like with things that have side effects, it's
the only way to know that we're even delivering the right answer; and
even when it's just an optimization problem, it's nice to give users a
way of fixing our planning failures that is better than asking them to
wait until we invent a way of improving the optimization decision in
some future release - which we may never even do.
I actually think that we should go "all in" here and allow the user to
specify either that they want materialization or that they don't want
materialization. If they specify neither, then we make some decision
which we may change in a future release. If they do specify
something, we do that.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
@ 2019-01-11 18:19 ` Andres Freund <andres@anarazel.de>
2019-01-11 18:58 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
1 sibling, 1 reply; 112+ messages in thread
From: Andres Freund @ 2019-01-11 18:19 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Thomas Munro <thomas.munro@enterprisedb.com>; Andreas Karlsson <andreas@proxel.se>; Andrew Gierth <andrew@tao11.riddles.org.uk>; Tom Lane <tgl@sss.pgh.pa.us>; David Fetter <david@fetter.org>; pgsql-hackers
On 2019-01-11 11:12:25 -0500, Robert Haas wrote:
> I actually think that we should go "all in" here and allow the user to
> specify either that they want materialization or that they don't want
> materialization. If they specify neither, then we make some decision
> which we may change in a future release. If they do specify
> something, we do that.
+many
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-11 18:19 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
@ 2019-01-11 18:58 ` Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 19:04 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 112+ messages in thread
From: Thomas Munro @ 2019-01-11 18:58 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Robert Haas <robertmhaas@gmail.com>; Andreas Karlsson <andreas@proxel.se>; Andrew Gierth <andrew@tao11.riddles.org.uk>; Tom Lane <tgl@sss.pgh.pa.us>; David Fetter <david@fetter.org>; pgsql-hackers
On Sat, Jan 12, 2019 at 7:19 AM Andres Freund <andres@anarazel.de> wrote:
> On 2019-01-11 11:12:25 -0500, Robert Haas wrote:
> > I actually think that we should go "all in" here and allow the user to
> > specify either that they want materialization or that they don't want
> > materialization. If they specify neither, then we make some decision
> > which we may change in a future release. If they do specify
> > something, we do that.
>
> +many
I think the syntax as proposed is almost OK if we only want to
grandfather in our historically hintful CTEs but discourage the
development of any future kinds of hints. Even then I don't love the
way it formalises a semi-procedural step at the same language level as
a glorious declarative relational query.
Maybe we could consider a more extensible syntax that is attached to
the contained SELECT rather than the containing WITH. Then CTEs would
be less special; there'd be a place to put hints controlling top-level
queries, subselects, views etc too (perhaps eventually join hints,
parallelism hints etc, but "materialize this" would be just another
one of those things). That'd be all-in.
--
Thomas Munro
http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-11 18:19 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 18:58 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
@ 2019-01-11 19:04 ` Andres Freund <andres@anarazel.de>
2019-01-11 19:10 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
0 siblings, 1 reply; 112+ messages in thread
From: Andres Freund @ 2019-01-11 19:04 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@enterprisedb.com>; +Cc: Robert Haas <robertmhaas@gmail.com>; Andreas Karlsson <andreas@proxel.se>; Andrew Gierth <andrew@tao11.riddles.org.uk>; Tom Lane <tgl@sss.pgh.pa.us>; David Fetter <david@fetter.org>; pgsql-hackers
Hi,
On 2019-01-12 07:58:39 +1300, Thomas Munro wrote:
> On Sat, Jan 12, 2019 at 7:19 AM Andres Freund <andres@anarazel.de> wrote:
> > On 2019-01-11 11:12:25 -0500, Robert Haas wrote:
> > > I actually think that we should go "all in" here and allow the user to
> > > specify either that they want materialization or that they don't want
> > > materialization. If they specify neither, then we make some decision
> > > which we may change in a future release. If they do specify
> > > something, we do that.
> >
> > +many
>
> I think the syntax as proposed is almost OK if we only want to
> grandfather in our historically hintful CTEs but discourage the
> development of any future kinds of hints. Even then I don't love the
> way it formalises a semi-procedural step at the same language level as
> a glorious declarative relational query.
>
> Maybe we could consider a more extensible syntax that is attached to
> the contained SELECT rather than the containing WITH. Then CTEs would
> be less special; there'd be a place to put hints controlling top-level
> queries, subselects, views etc too (perhaps eventually join hints,
> parallelism hints etc, but "materialize this" would be just another
> one of those things). That'd be all-in.
I think you have some purity arguments here, but the likelihood of us
developing a full-blown solution is not that high, and the lack of
inlinable CTEs is *really* hurting us. As long as the design doesn't
block a full solution, if we go there, I think it's a very acceptable
blemish in comparison to the benefits we'd get.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-11 18:19 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 18:58 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 19:04 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
@ 2019-01-11 19:10 ` Robert Haas <robertmhaas@gmail.com>
2019-01-11 19:39 ` Re: Early WIP/PoC for inlining CTEs Mike Rylander <mrylander@gmail.com>
2019-01-12 05:51 ` Re: Early WIP/PoC for inlining CTEs Pavel Stehule <pavel.stehule@gmail.com>
2019-01-17 15:48 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
0 siblings, 3 replies; 112+ messages in thread
From: Robert Haas @ 2019-01-11 19:10 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Thomas Munro <thomas.munro@enterprisedb.com>; Andreas Karlsson <andreas@proxel.se>; Andrew Gierth <andrew@tao11.riddles.org.uk>; Tom Lane <tgl@sss.pgh.pa.us>; David Fetter <david@fetter.org>; pgsql-hackers
On Fri, Jan 11, 2019 at 2:04 PM Andres Freund <andres@anarazel.de> wrote:
> > Maybe we could consider a more extensible syntax that is attached to
> > the contained SELECT rather than the containing WITH. Then CTEs would
> > be less special; there'd be a place to put hints controlling top-level
> > queries, subselects, views etc too (perhaps eventually join hints,
> > parallelism hints etc, but "materialize this" would be just another
> > one of those things). That'd be all-in.
>
> I think you have some purity arguments here, but the likelihood of us
> developing a full-blown solution is not that high, and the lack of
> inlinable CTEs is *really* hurting us. As long as the design doesn't
> block a full solution, if we go there, I think it's a very acceptable
> blemish in comparison to the benefits we'd get.
Also, it seems to me that this is properly a property of the
individual WITH clause, not the query as a whole.
I mean I suppose we could do
WITH or_with_out_you OPTIONS (materialized false) AS (SELECT 'mariah
carey') SELECT ...
That'd allow for extensibility, have the write scope, and look like
what we do elsewhere. It looks a little less elegant than
WITH cte_name [[NOT] MATERIALIZED] AS (query) main_query...
...but maybe elegance for extensibility is a good trade.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-11 18:19 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 18:58 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 19:04 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 19:10 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
@ 2019-01-11 19:39 ` Mike Rylander <mrylander@gmail.com>
2 siblings, 0 replies; 112+ messages in thread
From: Mike Rylander @ 2019-01-11 19:39 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; Thomas Munro <thomas.munro@enterprisedb.com>; Andreas Karlsson <andreas@proxel.se>; Andrew Gierth <andrew@tao11.riddles.org.uk>; Tom Lane <tgl@sss.pgh.pa.us>; David Fetter <david@fetter.org>; pgsql-hackers
On Fri, Jan 11, 2019 at 2:10 PM Robert Haas <robertmhaas@gmail.com> wrote:
>
> On Fri, Jan 11, 2019 at 2:04 PM Andres Freund <andres@anarazel.de> wrote:
> > > Maybe we could consider a more extensible syntax that is attached to
> > > the contained SELECT rather than the containing WITH. Then CTEs would
> > > be less special; there'd be a place to put hints controlling top-level
> > > queries, subselects, views etc too (perhaps eventually join hints,
> > > parallelism hints etc, but "materialize this" would be just another
> > > one of those things). That'd be all-in.
> >
> > I think you have some purity arguments here, but the likelihood of us
> > developing a full-blown solution is not that high, and the lack of
> > inlinable CTEs is *really* hurting us. As long as the design doesn't
> > block a full solution, if we go there, I think it's a very acceptable
> > blemish in comparison to the benefits we'd get.
>
> Also, it seems to me that this is properly a property of the
> individual WITH clause, not the query as a whole.
>
> I mean I suppose we could do
>
> WITH or_with_out_you OPTIONS (materialized false) AS (SELECT 'mariah
> carey') SELECT ...
>
> That'd allow for extensibility, have the write scope, and look like
> what we do elsewhere. It looks a little less elegant than
>
> WITH cte_name [[NOT] MATERIALIZED] AS (query) main_query...
>
> ...but maybe elegance for extensibility is a good trade.
>
Here, have $0.02 from the peanut gallery...
I mildly prefer the latter, elegant spelling, but if CTE inlining does
become a thing then I would /really/ want some way, any way, of
telling Postgres that I want it to materialize a particular CTE.
I use that currently-documented property of CTEs to structure large,
complicated OLAP queries on a regular basis, for performance.
Sometimes, such as when you have dozens of tables in a complex join
tree, breaking the query into logically related chunks (which I know
about, but the planner does not) via CTE is the only way to give the
planner a fighting chance of finding a good plan. Otherwise you get
stuck in the GEQO ghetto, or planning time is some non-trivial
multiple of execution time.
Thanks,
--
Mike Rylander
| Executive Director
| Equinox Open Library Initiative
| phone: 1-877-OPEN-ILS (673-6457)
| email: miker@equinoxinitiative.org
| web: http://equinoxinitiative.org
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-11 18:19 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 18:58 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 19:04 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 19:10 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
@ 2019-01-12 05:51 ` Pavel Stehule <pavel.stehule@gmail.com>
2 siblings, 0 replies; 112+ messages in thread
From: Pavel Stehule @ 2019-01-12 05:51 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; Thomas Munro <thomas.munro@enterprisedb.com>; Andreas Karlsson <andreas@proxel.se>; Andrew Gierth <andrew@tao11.riddles.org.uk>; Tom Lane <tgl@sss.pgh.pa.us>; David Fetter <david@fetter.org>; pgsql-hackers
pá 11. 1. 2019 v 20:11 odesílatel Robert Haas <robertmhaas@gmail.com>
napsal:
> On Fri, Jan 11, 2019 at 2:04 PM Andres Freund <andres@anarazel.de> wrote:
> > > Maybe we could consider a more extensible syntax that is attached to
> > > the contained SELECT rather than the containing WITH. Then CTEs would
> > > be less special; there'd be a place to put hints controlling top-level
> > > queries, subselects, views etc too (perhaps eventually join hints,
> > > parallelism hints etc, but "materialize this" would be just another
> > > one of those things). That'd be all-in.
> >
> > I think you have some purity arguments here, but the likelihood of us
> > developing a full-blown solution is not that high, and the lack of
> > inlinable CTEs is *really* hurting us. As long as the design doesn't
> > block a full solution, if we go there, I think it's a very acceptable
> > blemish in comparison to the benefits we'd get.
>
> Also, it seems to me that this is properly a property of the
> individual WITH clause, not the query as a whole.
>
> I mean I suppose we could do
>
> WITH or_with_out_you OPTIONS (materialized false) AS (SELECT 'mariah
> carey') SELECT ...
>
> That'd allow for extensibility, have the write scope, and look like
> what we do elsewhere. It looks a little less elegant than
>
> WITH cte_name [[NOT] MATERIALIZED] AS (query) main_query...
>
> ...but maybe elegance for extensibility is a good trade.
>
I like this explicit syntax (both variant can be used). From my
perspective, it is much better than hints in comments.
Regards
Pavel
>
> --
> Robert Haas
> EnterpriseDB: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
>
>
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-11 18:19 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 18:58 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 19:04 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 19:10 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
@ 2019-01-17 15:48 ` Andreas Karlsson <andreas@proxel.se>
2019-01-17 15:52 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-18 20:34 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2 siblings, 2 replies; 112+ messages in thread
From: Andreas Karlsson @ 2019-01-17 15:48 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; +Cc: Thomas Munro <thomas.munro@enterprisedb.com>; Andrew Gierth <andrew@tao11.riddles.org.uk>; Tom Lane <tgl@sss.pgh.pa.us>; David Fetter <david@fetter.org>; pgsql-hackers
On 1/11/19 8:10 PM, Robert Haas wrote:
> WITH cte_name [[NOT] MATERIALIZED] AS (query) main_query...
Hm, when would one want "NOT MATERIALIZED"? I am not sure I see the
usefulness of forcing inlining other than if we by default do not inline
when a CTE is referenced multiple times.
Do you imaging it working something like the below?
1. Default
# Not inlined
- Referenced multiple times
- Includes FOR UPDATE|SHARE
- Includes volatile functions
- Recurisve
- DML
# Inlined
- Simple case (no side effects, referenced once)
2. MATERIALIZED
# Not inlined
- Everything
# Inlined
- (none)
3. NOT MATERIALIZED
# Not inlined
- Recursive
- DML
# Inlined
- Everything else
Andreas
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-11 18:19 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 18:58 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 19:04 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 19:10 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-17 15:48 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
@ 2019-01-17 15:52 ` Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 0 replies; 112+ messages in thread
From: Tom Lane @ 2019-01-17 15:52 UTC (permalink / raw)
To: Andreas Karlsson <andreas@proxel.se>; +Cc: Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Thomas Munro <thomas.munro@enterprisedb.com>; Andrew Gierth <andrew@tao11.riddles.org.uk>; David Fetter <david@fetter.org>; pgsql-hackers
Andreas Karlsson <andreas@proxel.se> writes:
> On 1/11/19 8:10 PM, Robert Haas wrote:
>> WITH cte_name [[NOT] MATERIALIZED] AS (query) main_query...
> Hm, when would one want "NOT MATERIALIZED"? I am not sure I see the
> usefulness of forcing inlining other than if we by default do not inline
> when a CTE is referenced multiple times.
I'm also concerned about what we do if the user says NOT MATERIALIZED
but there are semantic or implementation reasons not to inline. Either
we throw an error or do something the user didn't expect, and neither
is very nice. So I'm not in favor of having that option.
regards, tom lane
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-11 18:19 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 18:58 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 19:04 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 19:10 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-17 15:48 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
@ 2019-01-18 20:34 ` Robert Haas <robertmhaas@gmail.com>
2019-01-18 20:42 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-21 13:40 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
1 sibling, 2 replies; 112+ messages in thread
From: Robert Haas @ 2019-01-18 20:34 UTC (permalink / raw)
To: Andreas Karlsson <andreas@proxel.se>; +Cc: Andres Freund <andres@anarazel.de>; Thomas Munro <thomas.munro@enterprisedb.com>; Andrew Gierth <andrew@tao11.riddles.org.uk>; Tom Lane <tgl@sss.pgh.pa.us>; David Fetter <david@fetter.org>; pgsql-hackers
On Thu, Jan 17, 2019 at 10:48 AM Andreas Karlsson <andreas@proxel.se> wrote:
> On 1/11/19 8:10 PM, Robert Haas wrote:
> > WITH cte_name [[NOT] MATERIALIZED] AS (query) main_query...
>
> Hm, when would one want "NOT MATERIALIZED"? I am not sure I see the
> usefulness of forcing inlining other than if we by default do not inline
> when a CTE is referenced multiple times.
When the planner materializes it, but the performance of the resulting
plan therefore sucks, I suppose.
I don't feel super-strongly about this, and Tom is right that there
may be cases where materialization is just not practical due to
implementation restrictions. But it's not crazy to imagine that
inlining a multiply-referenced CTE might create opportunities for
optimization at each of those places, perhaps not the same ones in
each case, whereas materializing it results in doing extra work.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-11 18:19 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 18:58 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 19:04 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 19:10 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-17 15:48 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-18 20:34 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
@ 2019-01-18 20:42 ` Andres Freund <andres@anarazel.de>
2019-01-18 20:44 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
1 sibling, 1 reply; 112+ messages in thread
From: Andres Freund @ 2019-01-18 20:42 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Andreas Karlsson <andreas@proxel.se>; Thomas Munro <thomas.munro@enterprisedb.com>; Andrew Gierth <andrew@tao11.riddles.org.uk>; Tom Lane <tgl@sss.pgh.pa.us>; David Fetter <david@fetter.org>; pgsql-hackers
On 2019-01-18 15:34:46 -0500, Robert Haas wrote:
> On Thu, Jan 17, 2019 at 10:48 AM Andreas Karlsson <andreas@proxel.se> wrote:
> > On 1/11/19 8:10 PM, Robert Haas wrote:
> > > WITH cte_name [[NOT] MATERIALIZED] AS (query) main_query...
> >
> > Hm, when would one want "NOT MATERIALIZED"? I am not sure I see the
> > usefulness of forcing inlining other than if we by default do not inline
> > when a CTE is referenced multiple times.
>
> When the planner materializes it, but the performance of the resulting
> plan therefore sucks, I suppose.
>
> I don't feel super-strongly about this, and Tom is right that there
> may be cases where materialization is just not practical due to
> implementation restrictions.
*not* materializing I assume?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-11 18:19 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 18:58 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 19:04 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 19:10 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-17 15:48 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-18 20:34 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-18 20:42 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
@ 2019-01-18 20:44 ` Robert Haas <robertmhaas@gmail.com>
0 siblings, 0 replies; 112+ messages in thread
From: Robert Haas @ 2019-01-18 20:44 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Andreas Karlsson <andreas@proxel.se>; Thomas Munro <thomas.munro@enterprisedb.com>; Andrew Gierth <andrew@tao11.riddles.org.uk>; Tom Lane <tgl@sss.pgh.pa.us>; David Fetter <david@fetter.org>; pgsql-hackers
On Fri, Jan 18, 2019 at 3:42 PM Andres Freund <andres@anarazel.de> wrote:
> On 2019-01-18 15:34:46 -0500, Robert Haas wrote:
> > On Thu, Jan 17, 2019 at 10:48 AM Andreas Karlsson <andreas@proxel.se> wrote:
> > > On 1/11/19 8:10 PM, Robert Haas wrote:
> > > > WITH cte_name [[NOT] MATERIALIZED] AS (query) main_query...
> > >
> > > Hm, when would one want "NOT MATERIALIZED"? I am not sure I see the
> > > usefulness of forcing inlining other than if we by default do not inline
> > > when a CTE is referenced multiple times.
> >
> > When the planner materializes it, but the performance of the resulting
> > plan therefore sucks, I suppose.
> >
> > I don't feel super-strongly about this, and Tom is right that there
> > may be cases where materialization is just not practical due to
> > implementation restrictions.
>
> *not* materializing I assume?
Right, sorry.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-11 18:19 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 18:58 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 19:04 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 19:10 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-17 15:48 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-18 20:34 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
@ 2019-01-21 13:40 ` Andreas Karlsson <andreas@proxel.se>
2019-01-21 15:27 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-22 05:13 ` Re: Early WIP/PoC for inlining CTEs Gavin Flower <GavinFlower@archidevsys.co.nz>
1 sibling, 2 replies; 112+ messages in thread
From: Andreas Karlsson @ 2019-01-21 13:40 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; Thomas Munro <thomas.munro@enterprisedb.com>; Andrew Gierth <andrew@tao11.riddles.org.uk>; Tom Lane <tgl@sss.pgh.pa.us>; David Fetter <david@fetter.org>; pgsql-hackers
On 1/18/19 9:34 PM, Robert Haas wrote:
> On Thu, Jan 17, 2019 at 10:48 AM Andreas Karlsson <andreas@proxel.se> wrote:
>> On 1/11/19 8:10 PM, Robert Haas wrote:
>>> WITH cte_name [[NOT] MATERIALIZED] AS (query) main_query...
>>
>> Hm, when would one want "NOT MATERIALIZED"? I am not sure I see the
>> usefulness of forcing inlining other than if we by default do not inline
>> when a CTE is referenced multiple times.
>
> When the planner materializes it, but the performance of the resulting
> plan therefore sucks, I suppose.
>
> I don't feel super-strongly about this, and Tom is right that there
> may be cases where materialization is just not practical due to
> implementation restrictions. But it's not crazy to imagine that
> inlining a multiply-referenced CTE might create opportunities for
> optimization at each of those places, perhaps not the same ones in
> each case, whereas materializing it results in doing extra work.
I see.
I have a minor biksheddish question about the syntax.
You proposed:
WITH cte_name [[NOT] MATERIALIZED] AS (query) main_query
While Andrew proposed:
WITH cte_name AS [[NOT] MATERIALIZED] (query) main_query
Do people have any preference between these two?
Andreas
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-11 18:19 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 18:58 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 19:04 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 19:10 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-17 15:48 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-18 20:34 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-21 13:40 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
@ 2019-01-21 15:27 ` Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 19:00 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
1 sibling, 1 reply; 112+ messages in thread
From: Tom Lane @ 2019-01-21 15:27 UTC (permalink / raw)
To: Andreas Karlsson <andreas@proxel.se>; +Cc: Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Thomas Munro <thomas.munro@enterprisedb.com>; Andrew Gierth <andrew@tao11.riddles.org.uk>; David Fetter <david@fetter.org>; pgsql-hackers
Andreas Karlsson <andreas@proxel.se> writes:
> I have a minor biksheddish question about the syntax.
> You proposed:
> WITH cte_name [[NOT] MATERIALIZED] AS (query) main_query
> While Andrew proposed:
> WITH cte_name AS [[NOT] MATERIALIZED] (query) main_query
> Do people have any preference between these two?
FWIW, I'd independently thought that the latter is more readable,
and probably less likely to have syntax problems with future
extensions (since AS is already fully reserved). Didn't get
around to mentioning it yet, but +1 for putting AS first.
regards, tom lane
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-11 18:19 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 18:58 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 19:04 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 19:10 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-17 15:48 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-18 20:34 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-21 13:40 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-21 15:27 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
@ 2019-01-28 19:00 ` Robert Haas <robertmhaas@gmail.com>
2019-01-28 20:35 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 112+ messages in thread
From: Robert Haas @ 2019-01-28 19:00 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Andreas Karlsson <andreas@proxel.se>; Andres Freund <andres@anarazel.de>; Thomas Munro <thomas.munro@enterprisedb.com>; Andrew Gierth <andrew@tao11.riddles.org.uk>; David Fetter <david@fetter.org>; pgsql-hackers
On Mon, Jan 21, 2019 at 10:28 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Andreas Karlsson <andreas@proxel.se> writes:
> > I have a minor biksheddish question about the syntax.
> > You proposed:
> > WITH cte_name [[NOT] MATERIALIZED] AS (query) main_query
> > While Andrew proposed:
> > WITH cte_name AS [[NOT] MATERIALIZED] (query) main_query
> > Do people have any preference between these two?
>
> FWIW, I'd independently thought that the latter is more readable,
> and probably less likely to have syntax problems with future
> extensions (since AS is already fully reserved). Didn't get
> around to mentioning it yet, but +1 for putting AS first.
It seems to me that as long as the query has to be surrounded by
non-optional parentheses and the options list never starts with a
parenthesis, there isn't much room for a grammar conflict either way.
If the query didn't have to be surrounded by parentheses, you would
want to put the options before the word AS so that the word AS would
serve as an unambiguous terminator for the options specification, but
if the query must be preceded by an opening parenthesis then as long
as the options list can't include an option that begins with such a
parenthesis we are in good shape.
However, generally we have not had great luck with just sticking
keywords in there (cf. VACUUM, ANALYZE, EXPLAIN, COPY) which is why I
suggested using a flexible syntax with parenthesized options. And
then you're going to have trouble getting bison to figure out what to
do after...
WITH something AS ( some_keyword
...because some_keyword might be the an option name or the first word
of a query. And putting that syntax before AS isn't any better,
because now it can be confused with a column name list.
I am not deeply worked up about this, just proposing some things to think about.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-11 18:19 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 18:58 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 19:04 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 19:10 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-17 15:48 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-18 20:34 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-21 13:40 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-21 15:27 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 19:00 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
@ 2019-01-28 20:35 ` Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 21:54 ` Re: Early WIP/PoC for inlining CTEs Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-01-28 22:05 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 2 replies; 112+ messages in thread
From: Tom Lane @ 2019-01-28 20:35 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Andreas Karlsson <andreas@proxel.se>; Andres Freund <andres@anarazel.de>; Thomas Munro <thomas.munro@enterprisedb.com>; Andrew Gierth <andrew@tao11.riddles.org.uk>; David Fetter <david@fetter.org>; pgsql-hackers
Robert Haas <robertmhaas@gmail.com> writes:
> However, generally we have not had great luck with just sticking
> keywords in there (cf. VACUUM, ANALYZE, EXPLAIN, COPY) which is why I
> suggested using a flexible syntax with parenthesized options.
Fair, except that as you then proceed to point out, that does not work
either before or after the AS.
Conceivably we could make it work without the parens:
WITH ctename AS [ option = value [ , .... ] ] ( query .... )
which for the immediate feature I'd be tempted to spell as
WITH ctename AS [ materialize = on/off ] ( query ... )
I think the only reason the syntax is MATERIALIZED with a D is that
that's already a keyword; it reads a bit awkwardly IMO. But if we
were accepting a ColId there, there'd be room to adjust the spelling.
That said, this is still clunkier than the existing proposal, and
I'm not really convinced that we need to leave room for more options.
regards, tom lane
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-11 18:19 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 18:58 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 19:04 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 19:10 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-17 15:48 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-18 20:34 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-21 13:40 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-21 15:27 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 19:00 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-28 20:35 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
@ 2019-01-28 21:54 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-01-29 02:36 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
1 sibling, 1 reply; 112+ messages in thread
From: Peter Eisentraut @ 2019-01-28 21:54 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; Robert Haas <robertmhaas@gmail.com>; +Cc: Andreas Karlsson <andreas@proxel.se>; Andres Freund <andres@anarazel.de>; Thomas Munro <thomas.munro@enterprisedb.com>; Andrew Gierth <andrew@tao11.riddles.org.uk>; David Fetter <david@fetter.org>; pgsql-hackers
On 28/01/2019 21:35, Tom Lane wrote:
> Conceivably we could make it work without the parens:
>
> WITH ctename AS [ option = value [ , .... ] ] ( query .... )
>
> which for the immediate feature I'd be tempted to spell as
>
> WITH ctename AS [ materialize = on/off ] ( query ... )
>
> I think the only reason the syntax is MATERIALIZED with a D is that
> that's already a keyword; it reads a bit awkwardly IMO. But if we
> were accepting a ColId there, there'd be room to adjust the spelling.
Or put it at the end?
WITH ctename AS ( query ) MATERIALIZED
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-11 18:19 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 18:58 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 19:04 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 19:10 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-17 15:48 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-18 20:34 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-21 13:40 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-21 15:27 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 19:00 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-28 20:35 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 21:54 ` Re: Early WIP/PoC for inlining CTEs Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
@ 2019-01-29 02:36 ` Andreas Karlsson <andreas@proxel.se>
0 siblings, 0 replies; 112+ messages in thread
From: Andreas Karlsson @ 2019-01-29 02:36 UTC (permalink / raw)
To: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Tom Lane <tgl@sss.pgh.pa.us>; Robert Haas <robertmhaas@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; Thomas Munro <thomas.munro@enterprisedb.com>; Andrew Gierth <andrew@tao11.riddles.org.uk>; David Fetter <david@fetter.org>; pgsql-hackers
On 1/28/19 10:54 PM, Peter Eisentraut wrote:
> Or put it at the end?
>
> WITH ctename AS ( query ) MATERIALIZED
Hm, seems like that would be easy to miss for long queries.
Andreas
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-11 18:19 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 18:58 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 19:04 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 19:10 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-17 15:48 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-18 20:34 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-21 13:40 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-21 15:27 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 19:00 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-28 20:35 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
@ 2019-01-28 22:05 ` Tom Lane <tgl@sss.pgh.pa.us>
2019-01-29 04:51 ` Re: Early WIP/PoC for inlining CTEs Michael Paquier <michael@paquier.xyz>
2019-02-03 11:16 ` Re: Early WIP/PoC for inlining CTEs Vik Fearing <vik.fearing@2ndquadrant.com>
2019-02-03 15:39 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 3 replies; 112+ messages in thread
From: Tom Lane @ 2019-01-28 22:05 UTC (permalink / raw)
To: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; +Cc: Robert Haas <robertmhaas@gmail.com>; Andreas Karlsson <andreas@proxel.se>; Andres Freund <andres@anarazel.de>; Thomas Munro <thomas.munro@enterprisedb.com>; Andrew Gierth <andrew@tao11.riddles.org.uk>; David Fetter <david@fetter.org>; pgsql-hackers
Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:
> On 28/01/2019 21:35, Tom Lane wrote:
>> Conceivably we could make it work without the parens:
>> ...
> Or put it at the end?
> WITH ctename AS ( query ) MATERIALIZED
Yeah, I thought about that too, but it doesn't seem like an improvement.
If the query is very long (which isn't unlikely) I think people would
prefer to see the option(s) up front.
regards, tom lane
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-11 18:19 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 18:58 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 19:04 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 19:10 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-17 15:48 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-18 20:34 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-21 13:40 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-21 15:27 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 19:00 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-28 20:35 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 22:05 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
@ 2019-01-29 04:51 ` Michael Paquier <michael@paquier.xyz>
2019-01-29 19:52 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2 siblings, 1 reply; 112+ messages in thread
From: Michael Paquier @ 2019-01-29 04:51 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Robert Haas <robertmhaas@gmail.com>; Andreas Karlsson <andreas@proxel.se>; Andres Freund <andres@anarazel.de>; Thomas Munro <thomas.munro@enterprisedb.com>; Andrew Gierth <andrew@tao11.riddles.org.uk>; David Fetter <david@fetter.org>; pgsql-hackers
On Mon, Jan 28, 2019 at 05:05:32PM -0500, Tom Lane wrote:
> Yeah, I thought about that too, but it doesn't seem like an improvement.
> If the query is very long (which isn't unlikely) I think people would
> prefer to see the option(s) up front.
Having these options at the front of the WITH clause looks more
natural to me.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../20190129045158.GD3121@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-11 18:19 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 18:58 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 19:04 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 19:10 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-17 15:48 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-18 20:34 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-21 13:40 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-21 15:27 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 19:00 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-28 20:35 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 22:05 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-29 04:51 ` Re: Early WIP/PoC for inlining CTEs Michael Paquier <michael@paquier.xyz>
@ 2019-01-29 19:52 ` Tom Lane <tgl@sss.pgh.pa.us>
2019-01-29 20:36 ` Re: Early WIP/PoC for inlining CTEs Merlin Moncure <mmoncure@gmail.com>
2019-01-30 02:35 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2019-02-02 19:01 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 3 replies; 112+ messages in thread
From: Tom Lane @ 2019-01-29 19:52 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Robert Haas <robertmhaas@gmail.com>; Andreas Karlsson <andreas@proxel.se>; Andres Freund <andres@anarazel.de>; Thomas Munro <thomas.munro@enterprisedb.com>; Andrew Gierth <andrew@tao11.riddles.org.uk>; David Fetter <david@fetter.org>; pgsql-hackers
Michael Paquier <michael@paquier.xyz> writes:
> On Mon, Jan 28, 2019 at 05:05:32PM -0500, Tom Lane wrote:
>> Yeah, I thought about that too, but it doesn't seem like an improvement.
>> If the query is very long (which isn't unlikely) I think people would
>> prefer to see the option(s) up front.
> Having these options at the front of the WITH clause looks more
> natural to me.
Well, we've managed to get agreement on the semantics of this thing,
let's not get hung up on the syntax details.
I propose that we implement and document this as
WITH ctename AS [ MATERIALIZE { ON | OFF } ] ( query )
which is maybe a bit clunky but not awful, and it would leave room
to generalize it to "AS [ optionname optionvalue [ , ... ] ]" if we
ever need to. Looking at the precedent of e.g. EXPLAIN, we could
probably allow just "MATERIALIZE" as well, with the boolean value
defaulting to true.
regards, tom lane
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-11 18:19 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 18:58 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 19:04 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 19:10 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-17 15:48 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-18 20:34 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-21 13:40 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-21 15:27 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 19:00 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-28 20:35 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 22:05 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-29 04:51 ` Re: Early WIP/PoC for inlining CTEs Michael Paquier <michael@paquier.xyz>
2019-01-29 19:52 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
@ 2019-01-29 20:36 ` Merlin Moncure <mmoncure@gmail.com>
2019-01-29 20:46 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-29 20:56 ` Re: Early WIP/PoC for inlining CTEs Chapman Flack <chap@anastigmatix.net>
2 siblings, 2 replies; 112+ messages in thread
From: Merlin Moncure @ 2019-01-29 20:36 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Robert Haas <robertmhaas@gmail.com>; Andreas Karlsson <andreas@proxel.se>; Andres Freund <andres@anarazel.de>; Thomas Munro <thomas.munro@enterprisedb.com>; Andrew Gierth <andrew@tao11.riddles.org.uk>; David Fetter <david@fetter.org>; pgsql-hackers
On Tue, Jan 29, 2019 at 1:53 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
> Michael Paquier <michael@paquier.xyz> writes:
> > On Mon, Jan 28, 2019 at 05:05:32PM -0500, Tom Lane wrote:
> >> Yeah, I thought about that too, but it doesn't seem like an improvement.
> >> If the query is very long (which isn't unlikely) I think people would
> >> prefer to see the option(s) up front.
>
> > Having these options at the front of the WITH clause looks more
> > natural to me.
>
> Well, we've managed to get agreement on the semantics of this thing,
> let's not get hung up on the syntax details.
>
> I propose that we implement and document this as
>
> WITH ctename AS [ MATERIALIZE { ON | OFF } ] ( query )
>
> which is maybe a bit clunky but not awful, and it would leave room
> to generalize it to "AS [ optionname optionvalue [ , ... ] ]" if we
> ever need to. Looking at the precedent of e.g. EXPLAIN, we could
> probably allow just "MATERIALIZE" as well, with the boolean value
> defaulting to true.
I hate to bikeshed here, but I think it's better english using that
style of syntax to say,
WITH ctename AS [ MATERIALIZATION { ON | OFF } ] ( query )
merlin
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-11 18:19 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 18:58 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 19:04 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 19:10 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-17 15:48 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-18 20:34 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-21 13:40 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-21 15:27 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 19:00 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-28 20:35 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 22:05 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-29 04:51 ` Re: Early WIP/PoC for inlining CTEs Michael Paquier <michael@paquier.xyz>
2019-01-29 19:52 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-29 20:36 ` Re: Early WIP/PoC for inlining CTEs Merlin Moncure <mmoncure@gmail.com>
@ 2019-01-29 20:46 ` Tom Lane <tgl@sss.pgh.pa.us>
2019-01-30 04:56 ` Re: Early WIP/PoC for inlining CTEs Merlin Moncure <mmoncure@gmail.com>
1 sibling, 1 reply; 112+ messages in thread
From: Tom Lane @ 2019-01-29 20:46 UTC (permalink / raw)
To: Merlin Moncure <mmoncure@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Robert Haas <robertmhaas@gmail.com>; Andreas Karlsson <andreas@proxel.se>; Andres Freund <andres@anarazel.de>; Thomas Munro <thomas.munro@enterprisedb.com>; Andrew Gierth <andrew@tao11.riddles.org.uk>; David Fetter <david@fetter.org>; pgsql-hackers
Merlin Moncure <mmoncure@gmail.com> writes:
> On Tue, Jan 29, 2019 at 1:53 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> I propose that we implement and document this as
>> WITH ctename AS [ MATERIALIZE { ON | OFF } ] ( query )
> I hate to bikeshed here, but I think it's better english using that
> style of syntax to say,
> WITH ctename AS [ MATERIALIZATION { ON | OFF } ] ( query )
Hm. Doesn't really seem to fit with our style for options elsewhere;
for instance, EXPLAIN's options are things like ANALYZE ON/OFF and
VERBOSE ON/OFF, not ANALYSIS and VERBOSITY.
regards, tom lane
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-11 18:19 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 18:58 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 19:04 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 19:10 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-17 15:48 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-18 20:34 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-21 13:40 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-21 15:27 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 19:00 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-28 20:35 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 22:05 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-29 04:51 ` Re: Early WIP/PoC for inlining CTEs Michael Paquier <michael@paquier.xyz>
2019-01-29 19:52 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-29 20:36 ` Re: Early WIP/PoC for inlining CTEs Merlin Moncure <mmoncure@gmail.com>
2019-01-29 20:46 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
@ 2019-01-30 04:56 ` Merlin Moncure <mmoncure@gmail.com>
0 siblings, 0 replies; 112+ messages in thread
From: Merlin Moncure @ 2019-01-30 04:56 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Robert Haas <robertmhaas@gmail.com>; Andreas Karlsson <andreas@proxel.se>; Andres Freund <andres@anarazel.de>; Thomas Munro <thomas.munro@enterprisedb.com>; Andrew Gierth <andrew@tao11.riddles.org.uk>; David Fetter <david@fetter.org>; pgsql-hackers
On Tuesday, January 29, 2019, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Merlin Moncure <mmoncure@gmail.com> writes:
> > On Tue, Jan 29, 2019 at 1:53 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> >> I propose that we implement and document this as
> >> WITH ctename AS [ MATERIALIZE { ON | OFF } ] ( query )
>
> > I hate to bikeshed here, but I think it's better english using that
> > style of syntax to say,
> > WITH ctename AS [ MATERIALIZATION { ON | OFF } ] ( query )
>
> Hm. Doesn't really seem to fit with our style for options elsewhere;
> for instance, EXPLAIN's options are things like ANALYZE ON/OFF and
> VERBOSE ON/OFF, not ANALYSIS and VERBOSITY.
>
> regards, tom lane
>
Yep...I'll concede the point.
merlin
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-11 18:19 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 18:58 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 19:04 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 19:10 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-17 15:48 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-18 20:34 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-21 13:40 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-21 15:27 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 19:00 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-28 20:35 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 22:05 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-29 04:51 ` Re: Early WIP/PoC for inlining CTEs Michael Paquier <michael@paquier.xyz>
2019-01-29 19:52 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-29 20:36 ` Re: Early WIP/PoC for inlining CTEs Merlin Moncure <mmoncure@gmail.com>
@ 2019-01-29 20:56 ` Chapman Flack <chap@anastigmatix.net>
1 sibling, 0 replies; 112+ messages in thread
From: Chapman Flack @ 2019-01-29 20:56 UTC (permalink / raw)
To: Merlin Moncure <mmoncure@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Robert Haas <robertmhaas@gmail.com>; Andreas Karlsson <andreas@proxel.se>; Andres Freund <andres@anarazel.de>; Thomas Munro <thomas.munro@enterprisedb.com>; Andrew Gierth <andrew@tao11.riddles.org.uk>; David Fetter <david@fetter.org>; pgsql-hackers
On 1/29/19 3:36 PM, Merlin Moncure wrote:
> I hate to bikeshed here, but I think it's better english using that
> style of syntax to say,
> WITH ctename AS [ MATERIALIZATION { ON | OFF } ] ( query )
I had been just about to also engage in bikeshedding, on grounds
that (to me) the MATERIALIZED/NOT MATERIALIZED form seemed more
natural:
FROM GROCER OBTAIN WAXED CUCUMBERS. (this seems downright natural)
FROM GROCER OBTAIN NOT WAXED CUCUMBERS. (nearly natural, s/NOT /UN/)
FROM GROCER OBTAIN WAX ON CUCUMBERS. (these read oddly to me)
FROM GROCER OBTAIN WAX OFF CUCUMBERS.
I do understand Tom's point that the wax-on/wax-off form generalizes
more easily to non-boolean future options. It would really read
better as a parenthetical, so too bad parentheses are already taken
to go around the query.
While gawking at the bikeshed, one more thing came to mind:
I like to hold out hope [1] that, one day, the WITH grammar could
be extended to handle lexically-scoped option settings like those in
the ISO standard.
It doesn't seem to me that any of these current proposals would get
in the way of that. Just another thing to have in mind.
Regards,
-Chap
[1]
https://wiki.postgresql.org/wiki/PostgreSQL_vs_SQL/XML_Standards#XMLBINARY_and_XMLNAMESPACES
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-11 18:19 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 18:58 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 19:04 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 19:10 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-17 15:48 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-18 20:34 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-21 13:40 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-21 15:27 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 19:00 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-28 20:35 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 22:05 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-29 04:51 ` Re: Early WIP/PoC for inlining CTEs Michael Paquier <michael@paquier.xyz>
2019-01-29 19:52 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
@ 2019-01-30 02:35 ` David Fetter <david@fetter.org>
2019-01-30 03:42 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2 siblings, 1 reply; 112+ messages in thread
From: David Fetter @ 2019-01-30 02:35 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Robert Haas <robertmhaas@gmail.com>; Andreas Karlsson <andreas@proxel.se>; Andres Freund <andres@anarazel.de>; Thomas Munro <thomas.munro@enterprisedb.com>; Andrew Gierth <andrew@tao11.riddles.org.uk>; pgsql-hackers
On Tue, Jan 29, 2019 at 02:52:44PM -0500, Tom Lane wrote:
> Michael Paquier <michael@paquier.xyz> writes:
> > On Mon, Jan 28, 2019 at 05:05:32PM -0500, Tom Lane wrote:
> >> Yeah, I thought about that too, but it doesn't seem like an improvement.
> >> If the query is very long (which isn't unlikely) I think people would
> >> prefer to see the option(s) up front.
>
> > Having these options at the front of the WITH clause looks more
> > natural to me.
>
> Well, we've managed to get agreement on the semantics of this thing,
> let's not get hung up on the syntax details.
>
> I propose that we implement and document this as
>
> WITH ctename AS [ MATERIALIZE { ON | OFF } ] ( query )
I think this would be better with parentheses like this:
WITH ctename [ ( MATERIALIZE { ON | OFF } ) ] AS ( query ) [, ... ]
and it's a lot easier to add more query hints later.
Best,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778
Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-11 18:19 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 18:58 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 19:04 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 19:10 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-17 15:48 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-18 20:34 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-21 13:40 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-21 15:27 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 19:00 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-28 20:35 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 22:05 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-29 04:51 ` Re: Early WIP/PoC for inlining CTEs Michael Paquier <michael@paquier.xyz>
2019-01-29 19:52 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-30 02:35 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
@ 2019-01-30 03:42 ` Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 0 replies; 112+ messages in thread
From: Tom Lane @ 2019-01-30 03:42 UTC (permalink / raw)
To: David Fetter <david@fetter.org>; +Cc: Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Robert Haas <robertmhaas@gmail.com>; Andreas Karlsson <andreas@proxel.se>; Andres Freund <andres@anarazel.de>; Thomas Munro <thomas.munro@enterprisedb.com>; Andrew Gierth <andrew@tao11.riddles.org.uk>; pgsql-hackers
David Fetter <david@fetter.org> writes:
> On Tue, Jan 29, 2019 at 02:52:44PM -0500, Tom Lane wrote:
>> I propose that we implement and document this as
>> WITH ctename AS [ MATERIALIZE { ON | OFF } ] ( query )
> I think this would be better with parentheses like this:
> WITH ctename [ ( MATERIALIZE { ON | OFF } ) ] AS ( query ) [, ... ]
I take it you haven't actually been reading this thread.
regards, tom lane
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-11 18:19 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 18:58 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 19:04 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 19:10 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-17 15:48 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-18 20:34 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-21 13:40 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-21 15:27 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 19:00 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-28 20:35 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 22:05 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-29 04:51 ` Re: Early WIP/PoC for inlining CTEs Michael Paquier <michael@paquier.xyz>
2019-01-29 19:52 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
@ 2019-02-02 19:01 ` Tom Lane <tgl@sss.pgh.pa.us>
2019-02-03 08:14 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-02-06 09:00 ` Re: Early WIP/PoC for inlining CTEs Bruce Momjian <bruce@momjian.us>
2 siblings, 2 replies; 112+ messages in thread
From: Tom Lane @ 2019-02-02 19:01 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Robert Haas <robertmhaas@gmail.com>; Andreas Karlsson <andreas@proxel.se>; Andres Freund <andres@anarazel.de>; Thomas Munro <thomas.munro@enterprisedb.com>; Andrew Gierth <andrew@tao11.riddles.org.uk>; David Fetter <david@fetter.org>; pgsql-hackers
I wrote:
> I propose that we implement and document this as
> WITH ctename AS [ MATERIALIZE { ON | OFF } ] ( query )
> which is maybe a bit clunky but not awful, and it would leave room
> to generalize it to "AS [ optionname optionvalue [ , ... ] ]" if we
> ever need to. Looking at the precedent of e.g. EXPLAIN, we could
> probably allow just "MATERIALIZE" as well, with the boolean value
> defaulting to true.
In hopes of moving things along, here's a version of the patch that
does it like that. This demonstrates that, in fact, we can accept
"keyword [value] [, ...]" style options without any parens and
there's no syntax conflict. We'd have to work a bit harder on the
actual code in gram.y if we wanted to handle multiple options,
but the Bison productions will work.
There's nothing particularly stopping us from accepting
"materialized" with a D in this syntax, instead of or in addition
to "materialize"; though I hesitate to mention it for fear of
another round of bikeshedding.
After further reflection I really don't like Andrew's suggestion
that we not document the rule that multiply-referenced CTEs won't
be inlined by default. That would be giving up the principle
that WITH calculations are not done multiple times by default,
and I draw the line at that. It's an often-useful behavior as
well as one that's been documented from day one, so I do not accept
the argument that we might someday override it on the basis of
nothing but planner cost estimates.
regards, tom lane
Attachments:
[text/x-diff] inlining-ctes-v11.patch (41.9K, ../../4591.1549134061@sss.pgh.pa.us/2-inlining-ctes-v11.patch)
download | inline diff:
diff --git a/contrib/pg_stat_statements/pg_stat_statements.c b/contrib/pg_stat_statements/pg_stat_statements.c
index ea2e4bc..9905593 100644
--- a/contrib/pg_stat_statements/pg_stat_statements.c
+++ b/contrib/pg_stat_statements/pg_stat_statements.c
@@ -2927,6 +2927,7 @@ JumbleExpr(pgssJumbleState *jstate, Node *node)
/* we store the string name because RTE_CTE RTEs need it */
APP_JUMB_STRING(cte->ctename);
+ APP_JUMB(cte->ctematerialized);
JumbleQuery(jstate, castNode(Query, cte->ctequery));
}
break;
diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out
index b3894d0..226ba56 100644
--- a/contrib/postgres_fdw/expected/postgres_fdw.out
+++ b/contrib/postgres_fdw/expected/postgres_fdw.out
@@ -1888,7 +1888,7 @@ SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) ORDER BY t1.c3, t
-- join in CTE
EXPLAIN (VERBOSE, COSTS OFF)
-WITH t (c1_1, c1_3, c2_1) AS (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
+WITH t (c1_1, c1_3, c2_1) AS MATERIALIZE ON (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
QUERY PLAN
-------------------------------------------------------------------------------------------------------------------------------------
Limit
@@ -1905,7 +1905,7 @@ WITH t (c1_1, c1_3, c2_1) AS (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2
Output: t.c1_1, t.c2_1, t.c1_3
(12 rows)
-WITH t (c1_1, c1_3, c2_1) AS (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
+WITH t (c1_1, c1_3, c2_1) AS MATERIALIZE ON (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
c1_1 | c2_1
------+------
101 | 101
diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql
index f438165..6399b8b 100644
--- a/contrib/postgres_fdw/sql/postgres_fdw.sql
+++ b/contrib/postgres_fdw/sql/postgres_fdw.sql
@@ -493,8 +493,8 @@ SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) ORDER BY t1.c3, t
SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) ORDER BY t1.c3, t1.c1 OFFSET 100 LIMIT 10 FOR SHARE;
-- join in CTE
EXPLAIN (VERBOSE, COSTS OFF)
-WITH t (c1_1, c1_3, c2_1) AS (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
-WITH t (c1_1, c1_3, c2_1) AS (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
+WITH t (c1_1, c1_3, c2_1) AS MATERIALIZE ON (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
+WITH t (c1_1, c1_3, c2_1) AS MATERIALIZE ON (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
-- ctid with whole-row reference
EXPLAIN (VERBOSE, COSTS OFF)
SELECT t1.ctid, t1, t2, t1.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) ORDER BY t1.c3, t1.c1 OFFSET 100 LIMIT 10;
diff --git a/doc/src/sgml/queries.sgml b/doc/src/sgml/queries.sgml
index 88bc189..0cd4056 100644
--- a/doc/src/sgml/queries.sgml
+++ b/doc/src/sgml/queries.sgml
@@ -2199,16 +2199,19 @@ SELECT n FROM t LIMIT 100;
</para>
<para>
- A useful property of <literal>WITH</literal> queries is that they are evaluated
- only once per execution of the parent query, even if they are referred to
- more than once by the parent query or sibling <literal>WITH</literal> queries.
+ A useful property of <literal>WITH</literal> queries is that they are
+ normally evaluated only once per execution of the parent query, even if
+ they are referred to more than once by the parent query or
+ sibling <literal>WITH</literal> queries.
Thus, expensive calculations that are needed in multiple places can be
placed within a <literal>WITH</literal> query to avoid redundant work. Another
possible application is to prevent unwanted multiple evaluations of
functions with side-effects.
- However, the other side of this coin is that the optimizer is less able to
- push restrictions from the parent query down into a <literal>WITH</literal> query
- than an ordinary subquery. The <literal>WITH</literal> query will generally be
+ However, the other side of this coin is that the optimizer is not able to
+ push restrictions from the parent query down into a multiply-referenced
+ <literal>WITH</literal> query, since that might affect all uses of the
+ <literal>WITH</literal> query's output when it should affect only one.
+ The multiply-referenced <literal>WITH</literal> query will be
evaluated as written, without suppression of rows that the parent query
might discard afterwards. (But, as mentioned above, evaluation might stop
early if the reference(s) to the query demand only a limited number of
@@ -2216,6 +2219,76 @@ SELECT n FROM t LIMIT 100;
</para>
<para>
+ However, if a <literal>WITH</literal> query is non-recursive and
+ side-effect-free (that is, it is a <literal>SELECT</literal> containing
+ no volatile functions) then it can be folded into the parent query,
+ allowing joint optimization of the two query levels. By default, this
+ happens if the parent query references the <literal>WITH</literal> query
+ just once, but not if it references the <literal>WITH</literal> query
+ more than once.
+ You can override the decision by specifying <literal>MATERIALIZE
+ ON</literal> to force separate calculation of the <literal>WITH</literal>
+ query, or by specifying <literal>MATERIALIZE OFF</literal> to force it to
+ be merged into the parent query.
+ The latter choice risks duplicate computation of
+ the <literal>WITH</literal> query, but it can still give a net savings if
+ each usage of the <literal>WITH</literal> query needs only a small part
+ of the <literal>WITH</literal> query's full output.
+ </para>
+
+ <para>
+ A simple example of these rules is
+<programlisting>
+WITH w AS (
+ SELECT * FROM big_table
+)
+SELECT * FROM w WHERE key = 123;
+</programlisting>
+ This <literal>WITH</literal> query will be folded, producing the same
+ execution plan as
+<programlisting>
+SELECT * FROM big_table WHERE key = 123;
+</programlisting>
+ In particular, if there's an index on <structfield>key</structfield>,
+ it will probably be used to fetch just the rows having <literal>key =
+ 123</literal>. On the other hand, in
+<programlisting>
+WITH w AS (
+ SELECT * FROM big_table
+)
+SELECT * FROM w AS w1 JOIN w AS w2 ON w1.key = w2.ref
+WHERE w2.key = 123;
+</programlisting>
+ the <literal>WITH</literal> query will be materialized, producing a
+ temporary copy of <structname>big_table</structname> that is then
+ joined with itself — without benefit of any index. This query
+ will be executed much more efficiently if written as
+<programlisting>
+WITH w AS MATERIALIZE OFF (
+ SELECT * FROM big_table
+)
+SELECT * FROM w AS w1 JOIN w AS w2 ON w1.key = w2.ref
+WHERE w2.key = 123;
+</programlisting>
+ so that the parent query's restrictions can be applied directly
+ to scans of <structname>big_table</structname>.
+ </para>
+
+ <para>
+ An example where <literal>MATERIALIZE OFF</literal> could be
+ undesirable is
+<programlisting>
+WITH w AS (
+ SELECT key, very_expensive_function(val) as f FROM some_table
+)
+SELECT * FROM w AS w1 JOIN w AS w2 ON w1.f = w2.f;
+</programlisting>
+ Here, materialization of the <literal>WITH</literal> query ensures
+ that <function>very_expensive_function</function> is evaluated only
+ once per table row, not twice.
+ </para>
+
+ <para>
The examples above only show <literal>WITH</literal> being used with
<command>SELECT</command>, but it can be attached in the same way to
<command>INSERT</command>, <command>UPDATE</command>, or <command>DELETE</command>.
diff --git a/doc/src/sgml/ref/select.sgml b/doc/src/sgml/ref/select.sgml
index 4db8142..6d29cf5 100644
--- a/doc/src/sgml/ref/select.sgml
+++ b/doc/src/sgml/ref/select.sgml
@@ -72,7 +72,7 @@ SELECT [ ALL | DISTINCT [ ON ( <replaceable class="parameter">expression</replac
<phrase>and <replaceable class="parameter">with_query</replaceable> is:</phrase>
- <replaceable class="parameter">with_query_name</replaceable> [ ( <replaceable class="parameter">column_name</replaceable> [, ...] ) ] AS ( <replaceable class="parameter">select</replaceable> | <replaceable class="parameter">values</replaceable> | <replaceable class="parameter">insert</replaceable> | <replaceable class="parameter">update</replaceable> | <replaceable class="parameter">delete</replaceable> )
+ <replaceable class="parameter">with_query_name</replaceable> [ ( <replaceable class="parameter">column_name</replaceable> [, ...] ) ] AS [ MATERIALIZE [ ON | OFF ] ] ( <replaceable class="parameter">select</replaceable> | <replaceable class="parameter">values</replaceable> | <replaceable class="parameter">insert</replaceable> | <replaceable class="parameter">update</replaceable> | <replaceable class="parameter">delete</replaceable> )
TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ]
</synopsis>
@@ -93,7 +93,8 @@ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ]
These effectively serve as temporary tables that can be referenced
in the <literal>FROM</literal> list. A <literal>WITH</literal> query
that is referenced more than once in <literal>FROM</literal> is
- computed only once.
+ computed only once,
+ unless specified otherwise with <literal>MATERIALIZE OFF</literal>.
(See <xref linkend="sql-with" endterm="sql-with-title"/> below.)
</para>
</listitem>
@@ -273,8 +274,17 @@ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ]
</para>
<para>
+ The primary query and the <literal>WITH</literal> queries are all
+ (notionally) executed at the same time. This implies that the effects of
+ a data-modifying statement in <literal>WITH</literal> cannot be seen from
+ other parts of the query, other than by reading its <literal>RETURNING</literal>
+ output. If two such data-modifying statements attempt to modify the same
+ row, the results are unspecified.
+ </para>
+
+ <para>
A key property of <literal>WITH</literal> queries is that they
- are evaluated only once per execution of the primary query,
+ are normally evaluated only once per execution of the primary query,
even if the primary query refers to them more than once.
In particular, data-modifying statements are guaranteed to be
executed once and only once, regardless of whether the primary query
@@ -282,12 +292,35 @@ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ]
</para>
<para>
- The primary query and the <literal>WITH</literal> queries are all
- (notionally) executed at the same time. This implies that the effects of
- a data-modifying statement in <literal>WITH</literal> cannot be seen from
- other parts of the query, other than by reading its <literal>RETURNING</literal>
- output. If two such data-modifying statements attempt to modify the same
- row, the results are unspecified.
+ However, a <literal>WITH</literal> query can be marked with
+ <literal>MATERIALIZE OFF</literal> to remove this guarantee. In that
+ case, the <literal>WITH</literal> query can be folded into the primary
+ query much as though it were a simple sub-<literal>SELECT</literal> in
+ the primary query's <literal>FROM</literal> clause. This results in
+ duplicate computations if the primary query refers to
+ that <literal>WITH</literal> query more than once; but if each such use
+ requires only a few rows of the <literal>WITH</literal> query's total
+ output, <literal>MATERIALIZE OFF</literal> can provide a net savings by
+ allowing the queries to be optimized jointly.
+ <literal>MATERIALIZE OFF</literal> is ignored if it is attached to
+ a <literal>WITH</literal> query that is recursive or is not
+ side-effect-free (i.e., is not a plain <literal>SELECT</literal>
+ containing no volatile functions).
+ </para>
+
+ <para>
+ By default, a side-effect-free <literal>WITH</literal> query is folded
+ into the primary query if it is used exactly once in the primary
+ query's <literal>FROM</literal> clause. This allows joint optimization
+ of the two query levels in situations where that should be semantically
+ invisible. However, such folding can be prevented by marking the
+ <literal>WITH</literal> query with <literal>MATERIALIZE [ ON ]</literal>.
+ That might be useful, for example, if the <literal>WITH</literal> query
+ is being used as an optimization fence to prevent the planner from
+ choosing a bad plan.
+ <productname>PostgreSQL</productname> versions before v12 never did
+ such folding, so queries written for older versions might rely on
+ <literal>WITH</literal> to act as an optimization fence.
</para>
<para>
@@ -2087,6 +2120,11 @@ SELECT distributors.* WHERE distributors.name = 'Westward';
<para>
<literal>ROWS FROM( ... )</literal> is an extension of the SQL standard.
</para>
+
+ <para>
+ The <literal>MATERIALIZE ON/OFF</literal> options
+ of <literal>WITH</literal> are extensions of the SQL standard.
+ </para>
</refsect2>
</refsect1>
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index b44ead2..e15724b 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2536,6 +2536,7 @@ _copyCommonTableExpr(const CommonTableExpr *from)
COPY_STRING_FIELD(ctename);
COPY_NODE_FIELD(aliascolnames);
+ COPY_SCALAR_FIELD(ctematerialized);
COPY_NODE_FIELD(ctequery);
COPY_LOCATION_FIELD(location);
COPY_SCALAR_FIELD(cterecursive);
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index 1e169e0..31499eb 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2791,6 +2791,7 @@ _equalCommonTableExpr(const CommonTableExpr *a, const CommonTableExpr *b)
{
COMPARE_STRING_FIELD(ctename);
COMPARE_NODE_FIELD(aliascolnames);
+ COMPARE_SCALAR_FIELD(ctematerialized);
COMPARE_NODE_FIELD(ctequery);
COMPARE_LOCATION_FIELD(location);
COMPARE_SCALAR_FIELD(cterecursive);
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index f97cf37..0f31fca 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2979,6 +2979,7 @@ _outCommonTableExpr(StringInfo str, const CommonTableExpr *node)
WRITE_STRING_FIELD(ctename);
WRITE_NODE_FIELD(aliascolnames);
+ WRITE_ENUM_FIELD(ctematerialized, CTEMaterialize);
WRITE_NODE_FIELD(ctequery);
WRITE_LOCATION_FIELD(location);
WRITE_BOOL_FIELD(cterecursive);
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index 3b00277..5aa4224 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -418,6 +418,7 @@ _readCommonTableExpr(void)
READ_STRING_FIELD(ctename);
READ_NODE_FIELD(aliascolnames);
+ READ_ENUM_FIELD(ctematerialized, CTEMaterialize);
READ_NODE_FIELD(ctequery);
READ_LOCATION_FIELD(location);
READ_BOOL_FIELD(cterecursive);
diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c
index b223972..5b0c90f 100644
--- a/src/backend/optimizer/plan/planner.c
+++ b/src/backend/optimizer/plan/planner.c
@@ -646,8 +646,8 @@ subquery_planner(PlannerGlobal *glob, Query *parse,
root->partColsUpdated = false;
/*
- * If there is a WITH list, process each WITH query and build an initplan
- * SubPlan structure for it.
+ * If there is a WITH list, process each WITH query and either convert it
+ * to RTE_SUBQUERY RTE(s) or build an initplan SubPlan structure for it.
*/
if (parse->cteList)
SS_process_ctes(root);
diff --git a/src/backend/optimizer/plan/subselect.c b/src/backend/optimizer/plan/subselect.c
index 915c6d0..c721054 100644
--- a/src/backend/optimizer/plan/subselect.c
+++ b/src/backend/optimizer/plan/subselect.c
@@ -57,6 +57,14 @@ typedef struct finalize_primnode_context
Bitmapset *paramids; /* Non-local PARAM_EXEC paramids found */
} finalize_primnode_context;
+typedef struct inline_cte_walker_context
+{
+ const char *ctename; /* name and relative level of target CTE */
+ int levelsup;
+ int refcount; /* number of remaining references */
+ Query *ctequery; /* query to substitute */
+} inline_cte_walker_context;
+
static Node *build_subplan(PlannerInfo *root, Plan *plan, PlannerInfo *subroot,
List *plan_params,
@@ -75,6 +83,10 @@ static Node *convert_testexpr_mutator(Node *node,
static bool subplan_is_hashable(Plan *plan);
static bool testexpr_is_hashable(Node *testexpr);
static bool hash_ok_operator(OpExpr *expr);
+static bool contain_dml(Node *node);
+static bool contain_dml_walker(Node *node, void *context);
+static void inline_cte(PlannerInfo *root, CommonTableExpr *cte);
+static bool inline_cte_walker(Node *node, inline_cte_walker_context *context);
static bool simplify_EXISTS_query(PlannerInfo *root, Query *query);
static Query *convert_EXISTS_to_ANY(PlannerInfo *root, Query *subselect,
Node **testexpr, List **paramIds);
@@ -804,10 +816,13 @@ hash_ok_operator(OpExpr *expr)
/*
* SS_process_ctes: process a query's WITH list
*
- * We plan each interesting WITH item and convert it to an initplan.
+ * Consider each CTE in the WITH list and either ignore it (if it's an
+ * unreferenced SELECT), "inline" it to create a regular sub-SELECT-in-FROM,
+ * or convert it to an initplan.
+ *
* A side effect is to fill in root->cte_plan_ids with a list that
* parallels root->parse->cteList and provides the subplan ID for
- * each CTE's initplan.
+ * each CTE's initplan, or a dummy ID (-1) if we didn't make an initplan.
*/
void
SS_process_ctes(PlannerInfo *root)
@@ -839,6 +854,44 @@ SS_process_ctes(PlannerInfo *root)
}
/*
+ * Consider inlining the CTE (creating RTE_SUBQUERY RTE(s)) instead of
+ * implementing it as a separately-planned CTE.
+ *
+ * We cannot inline if any of these conditions hold:
+ *
+ * 1. The user said not to (the CTEMaterializeAlways option).
+ *
+ * 2. The CTE is recursive.
+ *
+ * 3. The CTE has side-effects; this includes either not being a plain
+ * SELECT, or containing volatile functions. Inlining might change
+ * the side-effects, which would be bad.
+ *
+ * Otherwise, we have an option whether to inline or not. That should
+ * always be a win if there's just a single reference, but if the CTE
+ * is multiply-referenced then it's unclear: inlining adds duplicate
+ * computations, but the ability to absorb restrictions from the outer
+ * query level could outweigh that. We do not have nearly enough
+ * information at this point to tell whether that's true, so we let
+ * the user express a preference. Our default behavior is to inline
+ * only singly-referenced CTEs, but a CTE marked CTEMaterializeNever
+ * will be inlined even if multiply referenced.
+ */
+ if ((cte->ctematerialized == CTEMaterializeNever ||
+ (cte->ctematerialized == CTEMaterializeDefault &&
+ cte->cterefcount == 1)) &&
+ !cte->cterecursive &&
+ cmdType == CMD_SELECT &&
+ !contain_dml(cte->ctequery) &&
+ !contain_volatile_functions(cte->ctequery))
+ {
+ inline_cte(root, cte);
+ /* Make a dummy entry in cte_plan_ids */
+ root->cte_plan_ids = lappend_int(root->cte_plan_ids, -1);
+ continue;
+ }
+
+ /*
* Copy the source Query node. Probably not necessary, but let's keep
* this similar to make_subplan.
*/
@@ -935,6 +988,127 @@ SS_process_ctes(PlannerInfo *root)
}
/*
+ * contain_dml: is any subquery not a plain SELECT?
+ *
+ * We reject SELECT FOR UPDATE/SHARE as well as INSERT etc.
+ */
+static bool
+contain_dml(Node *node)
+{
+ return contain_dml_walker(node, NULL);
+}
+
+static bool
+contain_dml_walker(Node *node, void *context)
+{
+ if (node == NULL)
+ return false;
+ if (IsA(node, Query))
+ {
+ Query *query = (Query *) node;
+
+ if (query->commandType != CMD_SELECT ||
+ query->rowMarks != NIL)
+ return true;
+
+ return query_tree_walker(query, contain_dml_walker, context, 0);
+ }
+ return expression_tree_walker(node, contain_dml_walker, context);
+}
+
+/*
+ * inline_cte: convert RTE_CTE references to given CTE into RTE_SUBQUERYs
+ */
+static void
+inline_cte(PlannerInfo *root, CommonTableExpr *cte)
+{
+ struct inline_cte_walker_context context;
+
+ context.ctename = cte->ctename;
+ /* Start at levelsup = -1 because we'll immediately increment it */
+ context.levelsup = -1;
+ context.refcount = cte->cterefcount;
+ context.ctequery = castNode(Query, cte->ctequery);
+
+ (void) inline_cte_walker((Node *) root->parse, &context);
+
+ /* Assert we replaced all references */
+ Assert(context.refcount == 0);
+}
+
+static bool
+inline_cte_walker(Node *node, inline_cte_walker_context *context)
+{
+ if (node == NULL)
+ return false;
+ if (IsA(node, Query))
+ {
+ Query *query = (Query *) node;
+
+ context->levelsup++;
+
+ /*
+ * Visit the query's RTE nodes after their contents; otherwise
+ * query_tree_walker would descend into the newly inlined CTE query,
+ * which we don't want.
+ */
+ (void) query_tree_walker(query, inline_cte_walker, context,
+ QTW_EXAMINE_RTES_AFTER);
+
+ context->levelsup--;
+
+ return false;
+ }
+ else if (IsA(node, RangeTblEntry))
+ {
+ RangeTblEntry *rte = (RangeTblEntry *) node;
+
+ if (rte->rtekind == RTE_CTE &&
+ strcmp(rte->ctename, context->ctename) == 0 &&
+ rte->ctelevelsup == context->levelsup)
+ {
+ /*
+ * Found a reference to replace. Generate a copy of the CTE query
+ * with appropriate level adjustment for outer references (e.g.,
+ * to other CTEs).
+ */
+ Query *newquery = copyObject(context->ctequery);
+
+ if (context->levelsup > 0)
+ IncrementVarSublevelsUp((Node *) newquery, context->levelsup, 1);
+
+ /*
+ * Convert the RTE_CTE RTE into a RTE_SUBQUERY.
+ *
+ * Historically, a FOR UPDATE clause has been treated as extending
+ * into views and subqueries, but not into CTEs. We preserve this
+ * distinction by not trying to push rowmarks into the new
+ * subquery.
+ */
+ rte->rtekind = RTE_SUBQUERY;
+ rte->subquery = newquery;
+ rte->security_barrier = false;
+
+ /* Zero out CTE-specific fields */
+ rte->ctename = NULL;
+ rte->ctelevelsup = 0;
+ rte->self_reference = false;
+ rte->coltypes = NIL;
+ rte->coltypmods = NIL;
+ rte->colcollations = NIL;
+
+ /* Count the number of replacements we've done */
+ context->refcount--;
+ }
+
+ return false;
+ }
+
+ return expression_tree_walker(node, inline_cte_walker, context);
+}
+
+
+/*
* convert_ANY_sublink_to_join: try to convert an ANY SubLink to a join
*
* The caller has found an ANY SubLink at the top level of one of the query's
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index c1faf41..5f3745a 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -479,7 +479,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> row explicit_row implicit_row type_list array_expr_list
%type <node> case_expr case_arg when_clause case_default
%type <list> when_clause_list
-%type <ival> sub_type
+%type <ival> sub_type cte_opts
%type <value> NumericOnly
%type <list> NumericOnly_list
%type <alias> alias_clause opt_alias_clause
@@ -11291,6 +11291,8 @@ simple_select:
*
* We don't currently support the SEARCH or CYCLE clause.
*
+ * Postgres-specific options can be inserted after AS.
+ *
* Recognizing WITH_LA here allows a CTE to be named TIME or ORDINALITY.
*/
with_clause:
@@ -11322,17 +11324,50 @@ cte_list:
| cte_list ',' common_table_expr { $$ = lappend($1, $3); }
;
-common_table_expr: name opt_name_list AS '(' PreparableStmt ')'
+common_table_expr: name opt_name_list AS cte_opts '(' PreparableStmt ')'
{
CommonTableExpr *n = makeNode(CommonTableExpr);
n->ctename = $1;
n->aliascolnames = $2;
- n->ctequery = $5;
+ n->ctematerialized = $4;
+ n->ctequery = $6;
n->location = @1;
$$ = (Node *) n;
}
;
+/*
+ * In general, we could allow arbitrary options for a CTE; for the moment
+ * this piggybacks on EXPLAIN's option productions. Since only one option
+ * is actually supported, just reduce the list to an enum result immediately.
+ */
+cte_opts: explain_option_list
+ {
+ int matopt = CTEMaterializeDefault;
+ ListCell *option;
+ foreach(option, $1)
+ {
+ DefElem *defel = (DefElem *) lfirst(option);
+
+ if (strcmp(defel->defname, "materialize") == 0)
+ {
+ if (defGetBoolean(defel))
+ matopt = CTEMaterializeAlways;
+ else
+ matopt = CTEMaterializeNever;
+ }
+ else
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("unrecognized WITH option \"%s\"",
+ defel->defname),
+ parser_errposition(defel->location)));
+ }
+ $$ = matopt;
+ }
+ | /*EMPTY*/ { $$ = CTEMaterializeDefault; }
+ ;
+
opt_with_clause:
with_clause { $$ = $1; }
| /*EMPTY*/ { $$ = NULL; }
@@ -16214,6 +16249,7 @@ makeRecursiveViewSelect(char *relname, List *aliases, Node *query)
/* create common table expression */
cte->ctename = relname;
cte->aliascolnames = aliases;
+ cte->ctematerialized = CTEMaterializeDefault;
cte->ctequery = query;
cte->location = -1;
diff --git a/src/backend/utils/adt/ruleutils.c b/src/backend/utils/adt/ruleutils.c
index 17a28c2..4103602 100644
--- a/src/backend/utils/adt/ruleutils.c
+++ b/src/backend/utils/adt/ruleutils.c
@@ -5166,7 +5166,19 @@ get_with_clause(Query *query, deparse_context *context)
}
appendStringInfoChar(buf, ')');
}
- appendStringInfoString(buf, " AS (");
+ appendStringInfoString(buf, " AS ");
+ switch (cte->ctematerialized)
+ {
+ case CTEMaterializeDefault:
+ break;
+ case CTEMaterializeAlways:
+ appendStringInfoString(buf, "MATERIALIZE ON ");
+ break;
+ case CTEMaterializeNever:
+ appendStringInfoString(buf, "MATERIALIZE OFF ");
+ break;
+ }
+ appendStringInfoChar(buf, '(');
if (PRETTY_INDENT(context))
appendContextKeyword(context, "", 0, 0, 0);
get_query_def((Query *) cte->ctequery, buf, context->namespaces, NULL,
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 2fe14d7..14bea39 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -1402,11 +1402,19 @@ typedef struct OnConflictClause
*
* We don't currently support the SEARCH or CYCLE clause.
*/
+typedef enum CTEMaterialize
+{
+ CTEMaterializeDefault, /* no option specified */
+ CTEMaterializeAlways, /* MATERIALIZE ON */
+ CTEMaterializeNever /* MATERIALIZE OFF */
+} CTEMaterialize;
+
typedef struct CommonTableExpr
{
NodeTag type;
char *ctename; /* query name (never qualified) */
List *aliascolnames; /* optional list of column names */
+ CTEMaterialize ctematerialized; /* is this an optimization fence? */
/* SelectStmt/InsertStmt/etc before parse analysis, Query afterwards: */
Node *ctequery; /* the CTE's subquery */
int location; /* token location, or -1 if unknown */
diff --git a/src/test/regress/expected/rowsecurity.out b/src/test/regress/expected/rowsecurity.out
index 1d12b01..8971f88 100644
--- a/src/test/regress/expected/rowsecurity.out
+++ b/src/test/regress/expected/rowsecurity.out
@@ -2179,7 +2179,7 @@ EXPLAIN (COSTS OFF) EXECUTE plancache_test;
Filter: (((a % 2) = 0) AND f_leak(b))
(2 rows)
-PREPARE plancache_test2 AS WITH q AS (SELECT * FROM z1 WHERE f_leak(b)) SELECT * FROM q,z2;
+PREPARE plancache_test2 AS WITH q AS MATERIALIZE ON (SELECT * FROM z1 WHERE f_leak(b)) SELECT * FROM q,z2;
EXPLAIN (COSTS OFF) EXECUTE plancache_test2;
QUERY PLAN
-------------------------------------------------
@@ -2192,7 +2192,7 @@ EXPLAIN (COSTS OFF) EXECUTE plancache_test2;
-> Seq Scan on z2
(7 rows)
-PREPARE plancache_test3 AS WITH q AS (SELECT * FROM z2) SELECT * FROM q,z1 WHERE f_leak(z1.b);
+PREPARE plancache_test3 AS WITH q AS MATERIALIZE ON (SELECT * FROM z2) SELECT * FROM q,z1 WHERE f_leak(z1.b);
EXPLAIN (COSTS OFF) EXECUTE plancache_test3;
QUERY PLAN
-----------------------------------------------------
@@ -2826,7 +2826,7 @@ ALTER TABLE t1 ENABLE ROW LEVEL SECURITY;
GRANT ALL ON t1 TO regress_rls_bob;
INSERT INTO t1 (SELECT x, md5(x::text) FROM generate_series(0,20) x);
SET SESSION AUTHORIZATION regress_rls_bob;
-WITH cte1 AS (SELECT * FROM t1 WHERE f_leak(b)) SELECT * FROM cte1;
+WITH cte1 AS MATERIALIZE ON (SELECT * FROM t1 WHERE f_leak(b)) SELECT * FROM cte1;
NOTICE: f_leak => cfcd208495d565ef66e7dff9f98764da
NOTICE: f_leak => c81e728d9d4c2f636f067f89cc14862c
NOTICE: f_leak => a87ff679a2f3e71d9181a67b7542122c
@@ -2853,7 +2853,8 @@ NOTICE: f_leak => 98f13708210194c475687be6106a3b84
20 | 98f13708210194c475687be6106a3b84
(11 rows)
-EXPLAIN (COSTS OFF) WITH cte1 AS (SELECT * FROM t1 WHERE f_leak(b)) SELECT * FROM cte1;
+EXPLAIN (COSTS OFF)
+WITH cte1 AS MATERIALIZE ON (SELECT * FROM t1 WHERE f_leak(b)) SELECT * FROM cte1;
QUERY PLAN
-------------------------------------------------
CTE Scan on cte1
diff --git a/src/test/regress/expected/rowtypes.out b/src/test/regress/expected/rowtypes.out
index 054faabb..e02f99a 100644
--- a/src/test/regress/expected/rowtypes.out
+++ b/src/test/regress/expected/rowtypes.out
@@ -1048,7 +1048,7 @@ from (values (1,row(1,2)), (1,row(null,null)), (1,null),
(6 rows)
explain (verbose, costs off)
-with r(a,b) as
+with r(a,b) as materialize on
(values (1,row(1,2)), (1,row(null,null)), (1,null),
(null,row(1,2)), (null,row(null,null)), (null,null) )
select r, r is null as isnull, r is not null as isnotnull from r;
@@ -1061,7 +1061,7 @@ select r, r is null as isnull, r is not null as isnotnull from r;
Output: "*VALUES*".column1, "*VALUES*".column2
(5 rows)
-with r(a,b) as
+with r(a,b) as materialize on
(values (1,row(1,2)), (1,row(null,null)), (1,null),
(null,row(1,2)), (null,row(null,null)), (null,null) )
select r, r is null as isnull, r is not null as isnotnull from r;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 2c8e21b..2137abd 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -3110,7 +3110,7 @@ explain (costs off) INSERT INTO hats VALUES ('h8', 'forbidden') RETURNING *;
(5 rows)
-- ensure upserting into a rule, with a CTE (different offsets!) works
-WITH data(hat_name, hat_color) AS (
+WITH data(hat_name, hat_color) AS MATERIALIZE ON (
VALUES ('h8', 'green'),
('h9', 'blue'),
('h7', 'forbidden')
@@ -3124,7 +3124,8 @@ RETURNING *;
h9 | blue
(2 rows)
-EXPLAIN (costs off) WITH data(hat_name, hat_color) AS (
+EXPLAIN (costs off)
+WITH data(hat_name, hat_color) AS MATERIALIZE ON (
VALUES ('h8', 'green'),
('h9', 'blue'),
('h7', 'forbidden')
diff --git a/src/test/regress/expected/subselect.out b/src/test/regress/expected/subselect.out
index a54b4a5..a7ccbc1 100644
--- a/src/test/regress/expected/subselect.out
+++ b/src/test/regress/expected/subselect.out
@@ -1154,3 +1154,159 @@ fetch backward all in c1;
(2 rows)
commit;
+--
+-- Tests for CTE inlining behavior
+--
+-- Basic subquery that can be inlined
+explain (verbose, costs off)
+with x as (select * from (select f1 from subselect_tbl) ss)
+select * from x where f1 = 1;
+ QUERY PLAN
+----------------------------------
+ Seq Scan on public.subselect_tbl
+ Output: subselect_tbl.f1
+ Filter: (subselect_tbl.f1 = 1)
+(3 rows)
+
+-- Explicitly request materialization
+explain (verbose, costs off)
+with x as materialize on (select * from (select f1 from subselect_tbl) ss)
+select * from x where f1 = 1;
+ QUERY PLAN
+------------------------------------------
+ CTE Scan on x
+ Output: x.f1
+ Filter: (x.f1 = 1)
+ CTE x
+ -> Seq Scan on public.subselect_tbl
+ Output: subselect_tbl.f1
+(6 rows)
+
+-- Stable functions are safe to inline
+explain (verbose, costs off)
+with x as (select * from (select f1, now() from subselect_tbl) ss)
+select * from x where f1 = 1;
+ QUERY PLAN
+-----------------------------------
+ Seq Scan on public.subselect_tbl
+ Output: subselect_tbl.f1, now()
+ Filter: (subselect_tbl.f1 = 1)
+(3 rows)
+
+-- Volatile functions prevent inlining
+explain (verbose, costs off)
+with x as (select * from (select f1, random() from subselect_tbl) ss)
+select * from x where f1 = 1;
+ QUERY PLAN
+----------------------------------------------
+ CTE Scan on x
+ Output: x.f1, x.random
+ Filter: (x.f1 = 1)
+ CTE x
+ -> Seq Scan on public.subselect_tbl
+ Output: subselect_tbl.f1, random()
+(6 rows)
+
+-- SELECT FOR UPDATE cannot be inlined
+explain (verbose, costs off)
+with x as (select * from (select f1 from subselect_tbl for update) ss)
+select * from x where f1 = 1;
+ QUERY PLAN
+--------------------------------------------------------------------
+ CTE Scan on x
+ Output: x.f1
+ Filter: (x.f1 = 1)
+ CTE x
+ -> Subquery Scan on ss
+ Output: ss.f1
+ -> LockRows
+ Output: subselect_tbl.f1, subselect_tbl.ctid
+ -> Seq Scan on public.subselect_tbl
+ Output: subselect_tbl.f1, subselect_tbl.ctid
+(10 rows)
+
+-- Multiply-referenced CTEs are inlined only when requested
+explain (verbose, costs off)
+with x as (select * from (select f1, now() as n from subselect_tbl) ss)
+select * from x, x x2 where x.n = x2.n;
+ QUERY PLAN
+-------------------------------------------
+ Merge Join
+ Output: x.f1, x.n, x2.f1, x2.n
+ Merge Cond: (x.n = x2.n)
+ CTE x
+ -> Seq Scan on public.subselect_tbl
+ Output: subselect_tbl.f1, now()
+ -> Sort
+ Output: x.f1, x.n
+ Sort Key: x.n
+ -> CTE Scan on x
+ Output: x.f1, x.n
+ -> Sort
+ Output: x2.f1, x2.n
+ Sort Key: x2.n
+ -> CTE Scan on x x2
+ Output: x2.f1, x2.n
+(16 rows)
+
+explain (verbose, costs off)
+with x as materialize off (select * from (select f1, now() as n from subselect_tbl) ss)
+select * from x, x x2 where x.n = x2.n;
+ QUERY PLAN
+----------------------------------------------------------------------------
+ Result
+ Output: subselect_tbl.f1, now(), subselect_tbl_1.f1, now()
+ One-Time Filter: (now() = now())
+ -> Nested Loop
+ Output: subselect_tbl.f1, subselect_tbl_1.f1
+ -> Seq Scan on public.subselect_tbl
+ Output: subselect_tbl.f1, subselect_tbl.f2, subselect_tbl.f3
+ -> Materialize
+ Output: subselect_tbl_1.f1
+ -> Seq Scan on public.subselect_tbl subselect_tbl_1
+ Output: subselect_tbl_1.f1
+(11 rows)
+
+-- Check handling of outer references
+explain (verbose, costs off)
+with x as (select * from int4_tbl)
+select * from (with y as (select * from x) select * from y) ss;
+ QUERY PLAN
+-----------------------------
+ Seq Scan on public.int4_tbl
+ Output: int4_tbl.f1
+(2 rows)
+
+explain (verbose, costs off)
+with x as materialize on (select * from int4_tbl)
+select * from (with y as (select * from x) select * from y) ss;
+ QUERY PLAN
+-------------------------------------
+ CTE Scan on x
+ Output: x.f1
+ CTE x
+ -> Seq Scan on public.int4_tbl
+ Output: int4_tbl.f1
+(5 rows)
+
+-- Ensure that we inline the currect CTE when there are
+-- multiple CTEs with the same name
+explain (verbose, costs off)
+with x as (select 1 as y)
+select * from (with x as (select 2 as y) select * from x) ss;
+ QUERY PLAN
+-------------
+ Result
+ Output: 2
+(2 rows)
+
+-- Row marks are not pushed into CTEs
+explain (verbose, costs off)
+with x as (select * from subselect_tbl)
+select * from x for update;
+ QUERY PLAN
+----------------------------------------------------------------
+ Seq Scan on public.subselect_tbl
+ Output: subselect_tbl.f1, subselect_tbl.f2, subselect_tbl.f3
+(2 rows)
+
diff --git a/src/test/regress/sql/rowsecurity.sql b/src/test/regress/sql/rowsecurity.sql
index 38e9b38..589e13f 100644
--- a/src/test/regress/sql/rowsecurity.sql
+++ b/src/test/regress/sql/rowsecurity.sql
@@ -840,10 +840,10 @@ EXPLAIN (COSTS OFF) SELECT * FROM z1 WHERE f_leak(b);
PREPARE plancache_test AS SELECT * FROM z1 WHERE f_leak(b);
EXPLAIN (COSTS OFF) EXECUTE plancache_test;
-PREPARE plancache_test2 AS WITH q AS (SELECT * FROM z1 WHERE f_leak(b)) SELECT * FROM q,z2;
+PREPARE plancache_test2 AS WITH q AS MATERIALIZE ON (SELECT * FROM z1 WHERE f_leak(b)) SELECT * FROM q,z2;
EXPLAIN (COSTS OFF) EXECUTE plancache_test2;
-PREPARE plancache_test3 AS WITH q AS (SELECT * FROM z2) SELECT * FROM q,z1 WHERE f_leak(z1.b);
+PREPARE plancache_test3 AS WITH q AS MATERIALIZE ON (SELECT * FROM z2) SELECT * FROM q,z1 WHERE f_leak(z1.b);
EXPLAIN (COSTS OFF) EXECUTE plancache_test3;
SET ROLE regress_rls_group1;
@@ -1071,8 +1071,9 @@ INSERT INTO t1 (SELECT x, md5(x::text) FROM generate_series(0,20) x);
SET SESSION AUTHORIZATION regress_rls_bob;
-WITH cte1 AS (SELECT * FROM t1 WHERE f_leak(b)) SELECT * FROM cte1;
-EXPLAIN (COSTS OFF) WITH cte1 AS (SELECT * FROM t1 WHERE f_leak(b)) SELECT * FROM cte1;
+WITH cte1 AS MATERIALIZE ON (SELECT * FROM t1 WHERE f_leak(b)) SELECT * FROM cte1;
+EXPLAIN (COSTS OFF)
+WITH cte1 AS MATERIALIZE ON (SELECT * FROM t1 WHERE f_leak(b)) SELECT * FROM cte1;
WITH cte1 AS (UPDATE t1 SET a = a + 1 RETURNING *) SELECT * FROM cte1; --fail
WITH cte1 AS (UPDATE t1 SET a = a RETURNING *) SELECT * FROM cte1; --ok
diff --git a/src/test/regress/sql/rowtypes.sql b/src/test/regress/sql/rowtypes.sql
index 454d462..8eda763 100644
--- a/src/test/regress/sql/rowtypes.sql
+++ b/src/test/regress/sql/rowtypes.sql
@@ -434,12 +434,12 @@ from (values (1,row(1,2)), (1,row(null,null)), (1,null),
(null,row(1,2)), (null,row(null,null)), (null,null) ) r(a,b);
explain (verbose, costs off)
-with r(a,b) as
+with r(a,b) as materialize on
(values (1,row(1,2)), (1,row(null,null)), (1,null),
(null,row(1,2)), (null,row(null,null)), (null,null) )
select r, r is null as isnull, r is not null as isnotnull from r;
-with r(a,b) as
+with r(a,b) as materialize on
(values (1,row(1,2)), (1,row(null,null)), (1,null),
(null,row(1,2)), (null,row(null,null)), (null,null) )
select r, r is null as isnull, r is not null as isnotnull from r;
diff --git a/src/test/regress/sql/rules.sql b/src/test/regress/sql/rules.sql
index f4ee30e..dc7317c 100644
--- a/src/test/regress/sql/rules.sql
+++ b/src/test/regress/sql/rules.sql
@@ -1132,7 +1132,7 @@ SELECT tablename, rulename, definition FROM pg_rules
explain (costs off) INSERT INTO hats VALUES ('h8', 'forbidden') RETURNING *;
-- ensure upserting into a rule, with a CTE (different offsets!) works
-WITH data(hat_name, hat_color) AS (
+WITH data(hat_name, hat_color) AS MATERIALIZE ON (
VALUES ('h8', 'green'),
('h9', 'blue'),
('h7', 'forbidden')
@@ -1140,7 +1140,8 @@ WITH data(hat_name, hat_color) AS (
INSERT INTO hats
SELECT * FROM data
RETURNING *;
-EXPLAIN (costs off) WITH data(hat_name, hat_color) AS (
+EXPLAIN (costs off)
+WITH data(hat_name, hat_color) AS MATERIALIZE ON (
VALUES ('h8', 'green'),
('h9', 'blue'),
('h7', 'forbidden')
diff --git a/src/test/regress/sql/subselect.sql b/src/test/regress/sql/subselect.sql
index 843f511..336cf61 100644
--- a/src/test/regress/sql/subselect.sql
+++ b/src/test/regress/sql/subselect.sql
@@ -625,3 +625,61 @@ move forward all in c1;
fetch backward all in c1;
commit;
+
+--
+-- Tests for CTE inlining behavior
+--
+
+-- Basic subquery that can be inlined
+explain (verbose, costs off)
+with x as (select * from (select f1 from subselect_tbl) ss)
+select * from x where f1 = 1;
+
+-- Explicitly request materialization
+explain (verbose, costs off)
+with x as materialize on (select * from (select f1 from subselect_tbl) ss)
+select * from x where f1 = 1;
+
+-- Stable functions are safe to inline
+explain (verbose, costs off)
+with x as (select * from (select f1, now() from subselect_tbl) ss)
+select * from x where f1 = 1;
+
+-- Volatile functions prevent inlining
+explain (verbose, costs off)
+with x as (select * from (select f1, random() from subselect_tbl) ss)
+select * from x where f1 = 1;
+
+-- SELECT FOR UPDATE cannot be inlined
+explain (verbose, costs off)
+with x as (select * from (select f1 from subselect_tbl for update) ss)
+select * from x where f1 = 1;
+
+-- Multiply-referenced CTEs are inlined only when requested
+explain (verbose, costs off)
+with x as (select * from (select f1, now() as n from subselect_tbl) ss)
+select * from x, x x2 where x.n = x2.n;
+
+explain (verbose, costs off)
+with x as materialize off (select * from (select f1, now() as n from subselect_tbl) ss)
+select * from x, x x2 where x.n = x2.n;
+
+-- Check handling of outer references
+explain (verbose, costs off)
+with x as (select * from int4_tbl)
+select * from (with y as (select * from x) select * from y) ss;
+
+explain (verbose, costs off)
+with x as materialize on (select * from int4_tbl)
+select * from (with y as (select * from x) select * from y) ss;
+
+-- Ensure that we inline the currect CTE when there are
+-- multiple CTEs with the same name
+explain (verbose, costs off)
+with x as (select 1 as y)
+select * from (with x as (select 2 as y) select * from x) ss;
+
+-- Row marks are not pushed into CTEs
+explain (verbose, costs off)
+with x as (select * from subselect_tbl)
+select * from x for update;
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-11 18:19 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 18:58 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 19:04 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 19:10 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-17 15:48 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-18 20:34 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-21 13:40 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-21 15:27 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 19:00 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-28 20:35 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 22:05 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-29 04:51 ` Re: Early WIP/PoC for inlining CTEs Michael Paquier <michael@paquier.xyz>
2019-01-29 19:52 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-02 19:01 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
@ 2019-02-03 08:14 ` Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-02-09 20:52 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 1 reply; 112+ messages in thread
From: Andrew Gierth @ 2019-02-03 08:14 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Robert Haas <robertmhaas@gmail.com>; Andreas Karlsson <andreas@proxel.se>; Andres Freund <andres@anarazel.de>; Thomas Munro <thomas.munro@enterprisedb.com>; David Fetter <david@fetter.org>; pgsql-hackers
>>>>> "Tom" == Tom Lane <tgl@sss.pgh.pa.us> writes:
Tom> After further reflection I really don't like Andrew's suggestion
Tom> that we not document the rule that multiply-referenced CTEs won't
Tom> be inlined by default. That would be giving up the principle that
Tom> WITH calculations are not done multiple times by default, and I
Tom> draw the line at that. It's an often-useful behavior as well as
Tom> one that's been documented from day one, so I do not accept the
Tom> argument that we might someday override it on the basis of nothing
Tom> but planner cost estimates.
The case that springs to mind is when a CTE with grouping is then joined
multiple times in the main query with different conditions. If the
planner is able to deduce (e.g. via ECs) that restrictions on grouped
columns can be pushed into the CTE, then inlining the CTE multiple times
might be a significant win. But if that isn't possible, then inlining
multiple times might be a significant loss.
If such a query is made into a view, then (given your position) the
decision of whether to inline has to be made at view creation time,
which doesn't seem desirable (even if we have to put up with it for the
present).
--
Andrew (irc:RhodiumToad)
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-11 18:19 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 18:58 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 19:04 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 19:10 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-17 15:48 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-18 20:34 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-21 13:40 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-21 15:27 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 19:00 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-28 20:35 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 22:05 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-29 04:51 ` Re: Early WIP/PoC for inlining CTEs Michael Paquier <michael@paquier.xyz>
2019-01-29 19:52 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-02 19:01 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-03 08:14 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
@ 2019-02-09 20:52 ` Tom Lane <tgl@sss.pgh.pa.us>
2019-02-14 15:11 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 112+ messages in thread
From: Tom Lane @ 2019-02-09 20:52 UTC (permalink / raw)
To: Andrew Gierth <andrew@tao11.riddles.org.uk>; +Cc: Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Robert Haas <robertmhaas@gmail.com>; Andreas Karlsson <andreas@proxel.se>; Andres Freund <andres@anarazel.de>; Thomas Munro <thomas.munro@enterprisedb.com>; David Fetter <david@fetter.org>; pgsql-hackers
Andrew Gierth <andrew@tao11.riddles.org.uk> writes:
> "Tom" == Tom Lane <tgl@sss.pgh.pa.us> writes:
> Tom> After further reflection I really don't like Andrew's suggestion
> Tom> that we not document the rule that multiply-referenced CTEs won't
> Tom> be inlined by default. That would be giving up the principle that
> Tom> WITH calculations are not done multiple times by default, and I
> Tom> draw the line at that. It's an often-useful behavior as well as
> Tom> one that's been documented from day one, so I do not accept the
> Tom> argument that we might someday override it on the basis of nothing
> Tom> but planner cost estimates.
> The case that springs to mind is when a CTE with grouping is then joined
> multiple times in the main query with different conditions. If the
> planner is able to deduce (e.g. via ECs) that restrictions on grouped
> columns can be pushed into the CTE, then inlining the CTE multiple times
> might be a significant win. But if that isn't possible, then inlining
> multiple times might be a significant loss.
Sure, but this is exactly the sort of situation where we should offer
a way for the user to force either decision to be made. I think it's
very unlikely that we'll ever be in a position to make a realistic
cost-based decision for that. Actually planning it out both ways would
be horrendously expensive (and probably none too reliable anyway, given
how shaky ndistinct estimates tend to be); and we certainly don't have
enough info to make a smart choice without doing that.
regards, tom lane
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-11 18:19 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 18:58 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 19:04 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 19:10 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-17 15:48 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-18 20:34 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-21 13:40 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-21 15:27 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 19:00 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-28 20:35 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 22:05 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-29 04:51 ` Re: Early WIP/PoC for inlining CTEs Michael Paquier <michael@paquier.xyz>
2019-01-29 19:52 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-02 19:01 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-03 08:14 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-02-09 20:52 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
@ 2019-02-14 15:11 ` Tom Lane <tgl@sss.pgh.pa.us>
2019-02-14 15:22 ` Re: Early WIP/PoC for inlining CTEs Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-02-15 02:34 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
0 siblings, 2 replies; 112+ messages in thread
From: Tom Lane @ 2019-02-14 15:11 UTC (permalink / raw)
To: Andrew Gierth <andrew@tao11.riddles.org.uk>; +Cc: Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Robert Haas <robertmhaas@gmail.com>; Andreas Karlsson <andreas@proxel.se>; Andres Freund <andres@anarazel.de>; Thomas Munro <thomas.munro@enterprisedb.com>; David Fetter <david@fetter.org>; pgsql-hackers
... so, have we beaten this topic to death yet? Can we make a decision?
Personally, I'd be happy with either of the last two patch versions
I posted (that is, either AS [[NOT] MATERIALIZED] or
AS [MATERIALIZE [ON|OFF]] syntax). But we gotta pick something.
regards, tom lane
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-11 18:19 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 18:58 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 19:04 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 19:10 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-17 15:48 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-18 20:34 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-21 13:40 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-21 15:27 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 19:00 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-28 20:35 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 22:05 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-29 04:51 ` Re: Early WIP/PoC for inlining CTEs Michael Paquier <michael@paquier.xyz>
2019-01-29 19:52 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-02 19:01 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-03 08:14 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-02-09 20:52 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-14 15:11 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
@ 2019-02-14 15:22 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-02-14 16:02 ` Re: Early WIP/PoC for inlining CTEs Alvaro Herrera <alvherre@2ndquadrant.com>
1 sibling, 1 reply; 112+ messages in thread
From: Peter Eisentraut @ 2019-02-14 15:22 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; Andrew Gierth <andrew@tao11.riddles.org.uk>; +Cc: Michael Paquier <michael@paquier.xyz>; Robert Haas <robertmhaas@gmail.com>; Andreas Karlsson <andreas@proxel.se>; Andres Freund <andres@anarazel.de>; Thomas Munro <thomas.munro@enterprisedb.com>; David Fetter <david@fetter.org>; pgsql-hackers
On 14/02/2019 16:11, Tom Lane wrote:
> ... so, have we beaten this topic to death yet? Can we make a decision?
>
> Personally, I'd be happy with either of the last two patch versions
> I posted (that is, either AS [[NOT] MATERIALIZED] or
> AS [MATERIALIZE [ON|OFF]] syntax). But we gotta pick something.
If we're not really planning to add any more options, I'd register a
light vote for MATERIALIZED. It reads easier, seems more grammatically
correct, and uses an existing word.
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-11 18:19 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 18:58 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 19:04 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 19:10 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-17 15:48 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-18 20:34 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-21 13:40 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-21 15:27 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 19:00 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-28 20:35 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 22:05 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-29 04:51 ` Re: Early WIP/PoC for inlining CTEs Michael Paquier <michael@paquier.xyz>
2019-01-29 19:52 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-02 19:01 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-03 08:14 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-02-09 20:52 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-14 15:11 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-14 15:22 ` Re: Early WIP/PoC for inlining CTEs Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
@ 2019-02-14 16:02 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2019-02-14 19:22 ` Re: Early WIP/PoC for inlining CTEs Merlin Moncure <mmoncure@gmail.com>
0 siblings, 1 reply; 112+ messages in thread
From: Alvaro Herrera @ 2019-02-14 16:02 UTC (permalink / raw)
To: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Andrew Gierth <andrew@tao11.riddles.org.uk>; Michael Paquier <michael@paquier.xyz>; Robert Haas <robertmhaas@gmail.com>; Andreas Karlsson <andreas@proxel.se>; Andres Freund <andres@anarazel.de>; Thomas Munro <thomas.munro@enterprisedb.com>; David Fetter <david@fetter.org>; pgsql-hackers
On 2019-Feb-14, Peter Eisentraut wrote:
> On 14/02/2019 16:11, Tom Lane wrote:
> > ... so, have we beaten this topic to death yet? Can we make a decision?
> >
> > Personally, I'd be happy with either of the last two patch versions
> > I posted (that is, either AS [[NOT] MATERIALIZED] or
> > AS [MATERIALIZE [ON|OFF]] syntax). But we gotta pick something.
>
> If we're not really planning to add any more options, I'd register a
> light vote for MATERIALIZED. It reads easier, seems more grammatically
> correct, and uses an existing word.
+1 for MATERIALIZED, as I proposed in
https://postgr.es/m/20170503173305.fetj4tz7kd56tjlr@alvherre.pgsql
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-11 18:19 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 18:58 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 19:04 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 19:10 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-17 15:48 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-18 20:34 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-21 13:40 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-21 15:27 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 19:00 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-28 20:35 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 22:05 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-29 04:51 ` Re: Early WIP/PoC for inlining CTEs Michael Paquier <michael@paquier.xyz>
2019-01-29 19:52 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-02 19:01 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-03 08:14 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-02-09 20:52 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-14 15:11 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-14 15:22 ` Re: Early WIP/PoC for inlining CTEs Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-02-14 16:02 ` Re: Early WIP/PoC for inlining CTEs Alvaro Herrera <alvherre@2ndquadrant.com>
@ 2019-02-14 19:22 ` Merlin Moncure <mmoncure@gmail.com>
2019-02-15 22:36 ` Re: Early WIP/PoC for inlining CTEs Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 1 reply; 112+ messages in thread
From: Merlin Moncure @ 2019-02-14 19:22 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Tom Lane <tgl@sss.pgh.pa.us>; Andrew Gierth <andrew@tao11.riddles.org.uk>; Michael Paquier <michael@paquier.xyz>; Robert Haas <robertmhaas@gmail.com>; Andreas Karlsson <andreas@proxel.se>; Andres Freund <andres@anarazel.de>; Thomas Munro <thomas.munro@enterprisedb.com>; David Fetter <david@fetter.org>; pgsql-hackers
On Thu, Feb 14, 2019 at 10:02 AM Alvaro Herrera
<alvherre@2ndquadrant.com> wrote:
>
> On 2019-Feb-14, Peter Eisentraut wrote:
>
> > On 14/02/2019 16:11, Tom Lane wrote:
> > > ... so, have we beaten this topic to death yet? Can we make a decision?
> > >
> > > Personally, I'd be happy with either of the last two patch versions
> > > I posted (that is, either AS [[NOT] MATERIALIZED] or
> > > AS [MATERIALIZE [ON|OFF]] syntax). But we gotta pick something.
> >
> > If we're not really planning to add any more options, I'd register a
> > light vote for MATERIALIZED. It reads easier, seems more grammatically
> > correct, and uses an existing word.
>
> +1 for MATERIALIZED, as I proposed in
> https://postgr.es/m/20170503173305.fetj4tz7kd56tjlr@alvherre.pgsql
Seconded!
merlin
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-11 18:19 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 18:58 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 19:04 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 19:10 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-17 15:48 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-18 20:34 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-21 13:40 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-21 15:27 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 19:00 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-28 20:35 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 22:05 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-29 04:51 ` Re: Early WIP/PoC for inlining CTEs Michael Paquier <michael@paquier.xyz>
2019-01-29 19:52 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-02 19:01 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-03 08:14 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-02-09 20:52 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-14 15:11 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-14 15:22 ` Re: Early WIP/PoC for inlining CTEs Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-02-14 16:02 ` Re: Early WIP/PoC for inlining CTEs Alvaro Herrera <alvherre@2ndquadrant.com>
2019-02-14 19:22 ` Re: Early WIP/PoC for inlining CTEs Merlin Moncure <mmoncure@gmail.com>
@ 2019-02-15 22:36 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-02-16 21:14 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 112+ messages in thread
From: Tomas Vondra @ 2019-02-15 22:36 UTC (permalink / raw)
To: Merlin Moncure <mmoncure@gmail.com>; Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Tom Lane <tgl@sss.pgh.pa.us>; Andrew Gierth <andrew@tao11.riddles.org.uk>; Michael Paquier <michael@paquier.xyz>; Robert Haas <robertmhaas@gmail.com>; Andreas Karlsson <andreas@proxel.se>; Andres Freund <andres@anarazel.de>; Thomas Munro <thomas.munro@enterprisedb.com>; David Fetter <david@fetter.org>; pgsql-hackers
On 2/14/19 8:22 PM, Merlin Moncure wrote:
> On Thu, Feb 14, 2019 at 10:02 AM Alvaro Herrera
> <alvherre@2ndquadrant.com> wrote:
>>
>> On 2019-Feb-14, Peter Eisentraut wrote:
>>
>>> On 14/02/2019 16:11, Tom Lane wrote:
>>>> ... so, have we beaten this topic to death yet? Can we make a decision?
>>>>
>>>> Personally, I'd be happy with either of the last two patch versions
>>>> I posted (that is, either AS [[NOT] MATERIALIZED] or
>>>> AS [MATERIALIZE [ON|OFF]] syntax). But we gotta pick something.
>>>
>>> If we're not really planning to add any more options, I'd register a
>>> light vote for MATERIALIZED. It reads easier, seems more grammatically
>>> correct, and uses an existing word.
>>
>> +1 for MATERIALIZED, as I proposed in
>> https://postgr.es/m/20170503173305.fetj4tz7kd56tjlr@alvherre.pgsql
>
> Seconded!
>
+1 to MATERIALIZED too
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-11 18:19 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 18:58 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 19:04 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 19:10 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-17 15:48 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-18 20:34 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-21 13:40 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-21 15:27 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 19:00 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-28 20:35 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 22:05 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-29 04:51 ` Re: Early WIP/PoC for inlining CTEs Michael Paquier <michael@paquier.xyz>
2019-01-29 19:52 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-02 19:01 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-03 08:14 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-02-09 20:52 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-14 15:11 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-14 15:22 ` Re: Early WIP/PoC for inlining CTEs Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-02-14 16:02 ` Re: Early WIP/PoC for inlining CTEs Alvaro Herrera <alvherre@2ndquadrant.com>
2019-02-14 19:22 ` Re: Early WIP/PoC for inlining CTEs Merlin Moncure <mmoncure@gmail.com>
2019-02-15 22:36 ` Re: Early WIP/PoC for inlining CTEs Tomas Vondra <tomas.vondra@2ndquadrant.com>
@ 2019-02-16 21:14 ` Tom Lane <tgl@sss.pgh.pa.us>
2019-02-16 21:33 ` Re: Early WIP/PoC for inlining CTEs Jonathan S. Katz <jkatz@postgresql.org>
2019-02-17 00:34 ` Re: Early WIP/PoC for inlining CTEs Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-02-17 02:52 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
0 siblings, 3 replies; 112+ messages in thread
From: Tom Lane @ 2019-02-16 21:14 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Merlin Moncure <mmoncure@gmail.com>; Alvaro Herrera <alvherre@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Andrew Gierth <andrew@tao11.riddles.org.uk>; Michael Paquier <michael@paquier.xyz>; Robert Haas <robertmhaas@gmail.com>; Andreas Karlsson <andreas@proxel.se>; Andres Freund <andres@anarazel.de>; Thomas Munro <thomas.munro@enterprisedb.com>; David Fetter <david@fetter.org>; pgsql-hackers
Tomas Vondra <tomas.vondra@2ndquadrant.com> writes:
> On 2/14/19 8:22 PM, Merlin Moncure wrote:
>> On Thu, Feb 14, 2019 at 10:02 AM Alvaro Herrera
>> <alvherre@2ndquadrant.com> wrote:
>>> On 2019-Feb-14, Peter Eisentraut wrote:
>>>> If we're not really planning to add any more options, I'd register a
>>>> light vote for MATERIALIZED. It reads easier, seems more grammatically
>>>> correct, and uses an existing word.
>>> +1 for MATERIALIZED, as I proposed in
>>> https://postgr.es/m/20170503173305.fetj4tz7kd56tjlr@alvherre.pgsql
>> Seconded!
> +1 to MATERIALIZED too
I was expecting more controversy ... pushed that way.
regards, tom lane
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-11 18:19 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 18:58 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 19:04 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 19:10 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-17 15:48 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-18 20:34 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-21 13:40 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-21 15:27 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 19:00 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-28 20:35 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 22:05 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-29 04:51 ` Re: Early WIP/PoC for inlining CTEs Michael Paquier <michael@paquier.xyz>
2019-01-29 19:52 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-02 19:01 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-03 08:14 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-02-09 20:52 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-14 15:11 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-14 15:22 ` Re: Early WIP/PoC for inlining CTEs Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-02-14 16:02 ` Re: Early WIP/PoC for inlining CTEs Alvaro Herrera <alvherre@2ndquadrant.com>
2019-02-14 19:22 ` Re: Early WIP/PoC for inlining CTEs Merlin Moncure <mmoncure@gmail.com>
2019-02-15 22:36 ` Re: Early WIP/PoC for inlining CTEs Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-02-16 21:14 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
@ 2019-02-16 21:33 ` Jonathan S. Katz <jkatz@postgresql.org>
2 siblings, 0 replies; 112+ messages in thread
From: Jonathan S. Katz @ 2019-02-16 21:33 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Merlin Moncure <mmoncure@gmail.com>; Alvaro Herrera <alvherre@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Andrew Gierth <andrew@tao11.riddles.org.uk>; Michael Paquier <michael@paquier.xyz>; Robert Haas <robertmhaas@gmail.com>; Andreas Karlsson <andreas@proxel.se>; Andres Freund <andres@anarazel.de>; Thomas Munro <thomas.munro@enterprisedb.com>; David Fetter <david@fetter.org>; pgsql-hackers
On 2/16/19 4:14 PM, Tom Lane wrote:
> Tomas Vondra <tomas.vondra@2ndquadrant.com> writes:
>> On 2/14/19 8:22 PM, Merlin Moncure wrote:
>>> On Thu, Feb 14, 2019 at 10:02 AM Alvaro Herrera
>>> <alvherre@2ndquadrant.com> wrote:
>>>> On 2019-Feb-14, Peter Eisentraut wrote:
>>>>> If we're not really planning to add any more options, I'd register a
>>>>> light vote for MATERIALIZED. It reads easier, seems more grammatically
>>>>> correct, and uses an existing word.
>
>>>> +1 for MATERIALIZED, as I proposed in
>>>> https://postgr.es/m/20170503173305.fetj4tz7kd56tjlr@alvherre.pgsql
>
>>> Seconded!
>
>> +1 to MATERIALIZED too
>
> I was expecting more controversy ... pushed that way.
This is awesome. Thank you everyone for working on this!
Attachments:
[application/pgp-signature] signature.asc (832B, ../../7c9a902b-2465-a2fd-395a-59f4eba212de@postgresql.org/2-signature.asc)
download
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-11 18:19 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 18:58 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 19:04 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 19:10 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-17 15:48 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-18 20:34 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-21 13:40 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-21 15:27 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 19:00 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-28 20:35 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 22:05 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-29 04:51 ` Re: Early WIP/PoC for inlining CTEs Michael Paquier <michael@paquier.xyz>
2019-01-29 19:52 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-02 19:01 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-03 08:14 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-02-09 20:52 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-14 15:11 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-14 15:22 ` Re: Early WIP/PoC for inlining CTEs Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-02-14 16:02 ` Re: Early WIP/PoC for inlining CTEs Alvaro Herrera <alvherre@2ndquadrant.com>
2019-02-14 19:22 ` Re: Early WIP/PoC for inlining CTEs Merlin Moncure <mmoncure@gmail.com>
2019-02-15 22:36 ` Re: Early WIP/PoC for inlining CTEs Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-02-16 21:14 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
@ 2019-02-17 00:34 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2 siblings, 0 replies; 112+ messages in thread
From: Tomas Vondra @ 2019-02-17 00:34 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Merlin Moncure <mmoncure@gmail.com>; Alvaro Herrera <alvherre@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Andrew Gierth <andrew@tao11.riddles.org.uk>; Michael Paquier <michael@paquier.xyz>; Robert Haas <robertmhaas@gmail.com>; Andreas Karlsson <andreas@proxel.se>; Andres Freund <andres@anarazel.de>; Thomas Munro <thomas.munro@enterprisedb.com>; David Fetter <david@fetter.org>; pgsql-hackers
On 2/16/19 10:14 PM, Tom Lane wrote:
> Tomas Vondra <tomas.vondra@2ndquadrant.com> writes:
>> On 2/14/19 8:22 PM, Merlin Moncure wrote:
>>> On Thu, Feb 14, 2019 at 10:02 AM Alvaro Herrera
>>> <alvherre@2ndquadrant.com> wrote:
>>>> On 2019-Feb-14, Peter Eisentraut wrote:
>>>>> If we're not really planning to add any more options, I'd register a
>>>>> light vote for MATERIALIZED. It reads easier, seems more grammatically
>>>>> correct, and uses an existing word.
>
>>>> +1 for MATERIALIZED, as I proposed in
>>>> https://postgr.es/m/20170503173305.fetj4tz7kd56tjlr@alvherre.pgsql
>
>>> Seconded!
>
>> +1 to MATERIALIZED too
>
> I was expecting more controversy ... pushed that way.
>
As you wish. I withdraw my previous vote and I propose
AS [NOT] MATERIALIZED [yes|false]
;-)
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-11 18:19 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 18:58 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 19:04 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 19:10 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-17 15:48 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-18 20:34 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-21 13:40 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-21 15:27 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 19:00 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-28 20:35 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 22:05 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-29 04:51 ` Re: Early WIP/PoC for inlining CTEs Michael Paquier <michael@paquier.xyz>
2019-01-29 19:52 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-02 19:01 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-03 08:14 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-02-09 20:52 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-14 15:11 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-14 15:22 ` Re: Early WIP/PoC for inlining CTEs Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-02-14 16:02 ` Re: Early WIP/PoC for inlining CTEs Alvaro Herrera <alvherre@2ndquadrant.com>
2019-02-14 19:22 ` Re: Early WIP/PoC for inlining CTEs Merlin Moncure <mmoncure@gmail.com>
2019-02-15 22:36 ` Re: Early WIP/PoC for inlining CTEs Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-02-16 21:14 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
@ 2019-02-17 02:52 ` Andreas Karlsson <andreas@proxel.se>
2019-02-26 11:41 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2 siblings, 1 reply; 112+ messages in thread
From: Andreas Karlsson @ 2019-02-17 02:52 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Merlin Moncure <mmoncure@gmail.com>; Alvaro Herrera <alvherre@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Andrew Gierth <andrew@tao11.riddles.org.uk>; Michael Paquier <michael@paquier.xyz>; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Thomas Munro <thomas.munro@enterprisedb.com>; David Fetter <david@fetter.org>; pgsql-hackers
On 16/02/2019 22.14, Tom Lane wrote:
> I was expecting more controversy ... pushed that way.
Thanks! And thanks to everyone else who worked on this patch!
Andreas
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-11 18:19 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 18:58 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 19:04 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 19:10 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-17 15:48 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-18 20:34 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-21 13:40 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-21 15:27 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 19:00 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-28 20:35 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 22:05 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-29 04:51 ` Re: Early WIP/PoC for inlining CTEs Michael Paquier <michael@paquier.xyz>
2019-01-29 19:52 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-02 19:01 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-03 08:14 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-02-09 20:52 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-14 15:11 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-14 15:22 ` Re: Early WIP/PoC for inlining CTEs Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-02-14 16:02 ` Re: Early WIP/PoC for inlining CTEs Alvaro Herrera <alvherre@2ndquadrant.com>
2019-02-14 19:22 ` Re: Early WIP/PoC for inlining CTEs Merlin Moncure <mmoncure@gmail.com>
2019-02-15 22:36 ` Re: Early WIP/PoC for inlining CTEs Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-02-16 21:14 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-17 02:52 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
@ 2019-02-26 11:41 ` Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-02-26 14:51 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 112+ messages in thread
From: Andrew Gierth @ 2019-02-26 11:41 UTC (permalink / raw)
To: Andreas Karlsson <andreas@proxel.se>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Merlin Moncure <mmoncure@gmail.com>; Alvaro Herrera <alvherre@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Michael Paquier <michael@paquier.xyz>; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Thomas Munro <thomas.munro@enterprisedb.com>; David Fetter <david@fetter.org>; pgsql-hackers
So it turns out there's another case we have to check for, as reported
on IRC by Yaroslav Schekin (though I've simplified his query a little):
WITH RECURSIVE
x(a) AS ((VALUES ('a'),('b'))
UNION ALL
(WITH
z AS /*NOT*/ MATERIALIZED (SELECT a FROM x)
SELECT z.a || z1.a AS a FROM z CROSS JOIN z AS z1
WHERE length(z.a || z1.a) < 5))
SELECT * FROM x;
Here, uncommenting that NOT actually changes the result, from 22 rows to
4 rows, because we end up generating multiple worktable scans and the
recursion logic is not set up to handle that.
So what I think we need to do here is to forbid inlining if (a) the
refcount is greater than 1 and (b) the CTE in question contains,
recursively anywhere inside its rtable or the rtables of any of its
nested CTEs, a "self_reference" RTE. We only need to make this check if
we're somewhere (possibly nested) inside a recursive query, but I don't
know how practical it will be to test that condition at the point we do
inlining; doing it unconditionally will (I think) be harmless because
self_reference will not be set.
--
Andrew (irc:RhodiumToad)
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-11 18:19 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 18:58 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 19:04 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 19:10 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-17 15:48 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-18 20:34 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-21 13:40 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-21 15:27 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 19:00 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-28 20:35 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 22:05 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-29 04:51 ` Re: Early WIP/PoC for inlining CTEs Michael Paquier <michael@paquier.xyz>
2019-01-29 19:52 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-02 19:01 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-03 08:14 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-02-09 20:52 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-14 15:11 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-14 15:22 ` Re: Early WIP/PoC for inlining CTEs Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-02-14 16:02 ` Re: Early WIP/PoC for inlining CTEs Alvaro Herrera <alvherre@2ndquadrant.com>
2019-02-14 19:22 ` Re: Early WIP/PoC for inlining CTEs Merlin Moncure <mmoncure@gmail.com>
2019-02-15 22:36 ` Re: Early WIP/PoC for inlining CTEs Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-02-16 21:14 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-17 02:52 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-02-26 11:41 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
@ 2019-02-26 14:51 ` Tom Lane <tgl@sss.pgh.pa.us>
2019-02-27 01:11 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-04-09 19:50 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 2 replies; 112+ messages in thread
From: Tom Lane @ 2019-02-26 14:51 UTC (permalink / raw)
To: Andrew Gierth <andrew@tao11.riddles.org.uk>; +Cc: Andreas Karlsson <andreas@proxel.se>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Merlin Moncure <mmoncure@gmail.com>; Alvaro Herrera <alvherre@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Michael Paquier <michael@paquier.xyz>; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Thomas Munro <thomas.munro@enterprisedb.com>; David Fetter <david@fetter.org>; pgsql-hackers
Andrew Gierth <andrew@tao11.riddles.org.uk> writes:
> Here, uncommenting that NOT actually changes the result, from 22 rows to
> 4 rows, because we end up generating multiple worktable scans and the
> recursion logic is not set up to handle that.
Ugh.
> So what I think we need to do here is to forbid inlining if (a) the
> refcount is greater than 1 and (b) the CTE in question contains,
> recursively anywhere inside its rtable or the rtables of any of its
> nested CTEs, a "self_reference" RTE.
That's kind of "ugh" too: it sounds expensive, and doing it in a way
that doesn't produce false positives would be even more complicated.
Idle uncaffeinated speculation: is it practical to fix the restriction
about multiple worktable scans?
Also, I thought of a somewhat-related scenario that the code isn't
accounting for: you can break the restrictions about single evaluation
with nested WITHs, like
with x as not materialized (with y as materialized (select random() r) select * from y)
select * from x, x x1;
In this particular example, we're saved from computing random() twice
by the checks for volatile functions. But without that, y is inlined
and computed twice, e.g.
explain verbose with x as not materialized (with y as (select now() r) select * from y)
select * from x, x x1;
QUERY PLAN
------------------------------------------------
Nested Loop (cost=0.00..0.06 rows=1 width=16)
Output: (now()), (now())
-> Result (cost=0.00..0.01 rows=1 width=8)
Output: now()
-> Result (cost=0.00..0.01 rows=1 width=8)
Output: now()
(6 rows)
As a user I think I'd find that surprising, and bad if y were expensive.
Is it practical to inline the outer "x" level and still compute "y"
only once? If not, I think we need to disallow inlining anything
that contains a "with".
regards, tom lane
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-11 18:19 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 18:58 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 19:04 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 19:10 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-17 15:48 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-18 20:34 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-21 13:40 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-21 15:27 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 19:00 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-28 20:35 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 22:05 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-29 04:51 ` Re: Early WIP/PoC for inlining CTEs Michael Paquier <michael@paquier.xyz>
2019-01-29 19:52 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-02 19:01 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-03 08:14 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-02-09 20:52 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-14 15:11 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-14 15:22 ` Re: Early WIP/PoC for inlining CTEs Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-02-14 16:02 ` Re: Early WIP/PoC for inlining CTEs Alvaro Herrera <alvherre@2ndquadrant.com>
2019-02-14 19:22 ` Re: Early WIP/PoC for inlining CTEs Merlin Moncure <mmoncure@gmail.com>
2019-02-15 22:36 ` Re: Early WIP/PoC for inlining CTEs Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-02-16 21:14 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-17 02:52 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-02-26 11:41 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-02-26 14:51 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
@ 2019-02-27 01:11 ` Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-02-27 04:45 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 1 reply; 112+ messages in thread
From: Andrew Gierth @ 2019-02-27 01:11 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Andreas Karlsson <andreas@proxel.se>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Merlin Moncure <mmoncure@gmail.com>; Alvaro Herrera <alvherre@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Michael Paquier <michael@paquier.xyz>; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Thomas Munro <thomas.munro@enterprisedb.com>; David Fetter <david@fetter.org>; pgsql-hackers
>>>>> "Tom" == Tom Lane <tgl@sss.pgh.pa.us> writes:
Tom> Also, I thought of a somewhat-related scenario that the code isn't
Tom> accounting for: you can break the restrictions about single
Tom> evaluation with nested WITHs, like
I also thought about that. But what I thought about it on reflection
was: if the user explicitly wrote NOT MATERIALIZED, then we should
assume they mean it.
--
Andrew (irc:RhodiumToad)
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-11 18:19 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 18:58 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 19:04 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 19:10 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-17 15:48 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-18 20:34 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-21 13:40 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-21 15:27 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 19:00 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-28 20:35 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 22:05 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-29 04:51 ` Re: Early WIP/PoC for inlining CTEs Michael Paquier <michael@paquier.xyz>
2019-01-29 19:52 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-02 19:01 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-03 08:14 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-02-09 20:52 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-14 15:11 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-14 15:22 ` Re: Early WIP/PoC for inlining CTEs Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-02-14 16:02 ` Re: Early WIP/PoC for inlining CTEs Alvaro Herrera <alvherre@2ndquadrant.com>
2019-02-14 19:22 ` Re: Early WIP/PoC for inlining CTEs Merlin Moncure <mmoncure@gmail.com>
2019-02-15 22:36 ` Re: Early WIP/PoC for inlining CTEs Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-02-16 21:14 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-17 02:52 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-02-26 11:41 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-02-26 14:51 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-27 01:11 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
@ 2019-02-27 04:45 ` Tom Lane <tgl@sss.pgh.pa.us>
2019-02-27 05:16 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
0 siblings, 1 reply; 112+ messages in thread
From: Tom Lane @ 2019-02-27 04:45 UTC (permalink / raw)
To: Andrew Gierth <andrew@tao11.riddles.org.uk>; +Cc: Andreas Karlsson <andreas@proxel.se>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Merlin Moncure <mmoncure@gmail.com>; Alvaro Herrera <alvherre@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Michael Paquier <michael@paquier.xyz>; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Thomas Munro <thomas.munro@enterprisedb.com>; David Fetter <david@fetter.org>; pgsql-hackers
Andrew Gierth <andrew@tao11.riddles.org.uk> writes:
> "Tom" == Tom Lane <tgl@sss.pgh.pa.us> writes:
> Tom> Also, I thought of a somewhat-related scenario that the code isn't
> Tom> accounting for: you can break the restrictions about single
> Tom> evaluation with nested WITHs, like
> I also thought about that. But what I thought about it on reflection
> was: if the user explicitly wrote NOT MATERIALIZED, then we should
> assume they mean it.
Ah, but the example I gave also had MATERIALIZED on the inner WITH.
Why should the user not also mean that?
regards, tom lane
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-11 18:19 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 18:58 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 19:04 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 19:10 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-17 15:48 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-18 20:34 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-21 13:40 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-21 15:27 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 19:00 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-28 20:35 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 22:05 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-29 04:51 ` Re: Early WIP/PoC for inlining CTEs Michael Paquier <michael@paquier.xyz>
2019-01-29 19:52 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-02 19:01 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-03 08:14 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-02-09 20:52 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-14 15:11 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-14 15:22 ` Re: Early WIP/PoC for inlining CTEs Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-02-14 16:02 ` Re: Early WIP/PoC for inlining CTEs Alvaro Herrera <alvherre@2ndquadrant.com>
2019-02-14 19:22 ` Re: Early WIP/PoC for inlining CTEs Merlin Moncure <mmoncure@gmail.com>
2019-02-15 22:36 ` Re: Early WIP/PoC for inlining CTEs Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-02-16 21:14 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-17 02:52 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-02-26 11:41 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-02-26 14:51 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-27 01:11 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-02-27 04:45 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
@ 2019-02-27 05:16 ` Andrew Gierth <andrew@tao11.riddles.org.uk>
0 siblings, 0 replies; 112+ messages in thread
From: Andrew Gierth @ 2019-02-27 05:16 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Andreas Karlsson <andreas@proxel.se>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Merlin Moncure <mmoncure@gmail.com>; Alvaro Herrera <alvherre@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Michael Paquier <michael@paquier.xyz>; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Thomas Munro <thomas.munro@enterprisedb.com>; David Fetter <david@fetter.org>; pgsql-hackers
>>>>> "Tom" == Tom Lane <tgl@sss.pgh.pa.us> writes:
>> I also thought about that. But what I thought about it on reflection
>> was: if the user explicitly wrote NOT MATERIALIZED, then we should
>> assume they mean it.
Tom> Ah, but the example I gave also had MATERIALIZED on the inner WITH.
Tom> Why should the user not also mean that?
The inner WITH does get materialized, it just gets materialized twice.
If the user doesn't want that, then they can avoid using NOT MATERIALIZED
on the outer CTE; but if we force it to materialize the outer query,
then that leaves the user without recourse.
Consider a case like:
create view foo as
with s as materialized (select something)
select * from large l
where l.foo in (select * from s) or l.bar in (select * from s);
with
bar as not materialized (select * from foo)
select * from bar b1, bar b2 where b1.col='x' and b2.col='y';
In a case like this, materializing "s" twice may be far less expensive
than materializing the result of "select * from large..." without
benefit of pushed-down quals.
--
Andrew (irc:RhodiumToad)
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-11 18:19 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 18:58 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 19:04 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 19:10 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-17 15:48 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-18 20:34 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-21 13:40 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-21 15:27 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 19:00 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-28 20:35 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 22:05 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-29 04:51 ` Re: Early WIP/PoC for inlining CTEs Michael Paquier <michael@paquier.xyz>
2019-01-29 19:52 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-02 19:01 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-03 08:14 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-02-09 20:52 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-14 15:11 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-14 15:22 ` Re: Early WIP/PoC for inlining CTEs Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-02-14 16:02 ` Re: Early WIP/PoC for inlining CTEs Alvaro Herrera <alvherre@2ndquadrant.com>
2019-02-14 19:22 ` Re: Early WIP/PoC for inlining CTEs Merlin Moncure <mmoncure@gmail.com>
2019-02-15 22:36 ` Re: Early WIP/PoC for inlining CTEs Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-02-16 21:14 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-17 02:52 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-02-26 11:41 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-02-26 14:51 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
@ 2019-04-09 19:50 ` Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 0 replies; 112+ messages in thread
From: Tom Lane @ 2019-04-09 19:50 UTC (permalink / raw)
To: Andrew Gierth <andrew@tao11.riddles.org.uk>; +Cc: Andreas Karlsson <andreas@proxel.se>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Merlin Moncure <mmoncure@gmail.com>; Alvaro Herrera <alvherre@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Michael Paquier <michael@paquier.xyz>; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Thomas Munro <thomas.munro@enterprisedb.com>; David Fetter <david@fetter.org>; pgsql-hackers
I wrote:
> Andrew Gierth <andrew@tao11.riddles.org.uk> writes:
>> So what I think we need to do here is to forbid inlining if (a) the
>> refcount is greater than 1 and (b) the CTE in question contains,
>> recursively anywhere inside its rtable or the rtables of any of its
>> nested CTEs, a "self_reference" RTE.
> That's kind of "ugh" too: it sounds expensive, and doing it in a way
> that doesn't produce false positives would be even more complicated.
After further investigation, I concluded that that wasn't that awful,
so done that way.
I'm still not entirely convinced about the behavior for nested WITHs
with different materialization specifications, but that seems like
a separate topic.
regards, tom lane
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-11 18:19 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 18:58 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 19:04 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 19:10 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-17 15:48 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-18 20:34 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-21 13:40 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-21 15:27 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 19:00 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-28 20:35 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 22:05 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-29 04:51 ` Re: Early WIP/PoC for inlining CTEs Michael Paquier <michael@paquier.xyz>
2019-01-29 19:52 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-02 19:01 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-03 08:14 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-02-09 20:52 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-14 15:11 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
@ 2019-02-15 02:34 ` Andreas Karlsson <andreas@proxel.se>
1 sibling, 0 replies; 112+ messages in thread
From: Andreas Karlsson @ 2019-02-15 02:34 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; Andrew Gierth <andrew@tao11.riddles.org.uk>; +Cc: Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Thomas Munro <thomas.munro@enterprisedb.com>; David Fetter <david@fetter.org>; pgsql-hackers
On 14/02/2019 16.11, Tom Lane wrote:
> ... so, have we beaten this topic to death yet? Can we make a decision?
>
> Personally, I'd be happy with either of the last two patch versions
> I posted (that is, either AS [[NOT] MATERIALIZED] or
> AS [MATERIALIZE [ON|OFF]] syntax). But we gotta pick something.
I am happy with either of those.
Andreas
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-11 18:19 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 18:58 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 19:04 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 19:10 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-17 15:48 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-18 20:34 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-21 13:40 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-21 15:27 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 19:00 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-28 20:35 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 22:05 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-29 04:51 ` Re: Early WIP/PoC for inlining CTEs Michael Paquier <michael@paquier.xyz>
2019-01-29 19:52 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-02 19:01 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
@ 2019-02-06 09:00 ` Bruce Momjian <bruce@momjian.us>
2019-02-14 15:25 ` Re: Early WIP/PoC for inlining CTEs Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
1 sibling, 1 reply; 112+ messages in thread
From: Bruce Momjian @ 2019-02-06 09:00 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Robert Haas <robertmhaas@gmail.com>; Andreas Karlsson <andreas@proxel.se>; Andres Freund <andres@anarazel.de>; Thomas Munro <thomas.munro@enterprisedb.com>; Andrew Gierth <andrew@tao11.riddles.org.uk>; David Fetter <david@fetter.org>; pgsql-hackers
On Sat, Feb 2, 2019 at 02:01:01PM -0500, Tom Lane wrote:
> I wrote:
> > I propose that we implement and document this as
> > WITH ctename AS [ MATERIALIZE { ON | OFF } ] ( query )
> > which is maybe a bit clunky but not awful, and it would leave room
> > to generalize it to "AS [ optionname optionvalue [ , ... ] ]" if we
> > ever need to. Looking at the precedent of e.g. EXPLAIN, we could
> > probably allow just "MATERIALIZE" as well, with the boolean value
> > defaulting to true.
>
> In hopes of moving things along, here's a version of the patch that
> does it like that. This demonstrates that, in fact, we can accept
> "keyword [value] [, ...]" style options without any parens and
> there's no syntax conflict. We'd have to work a bit harder on the
> actual code in gram.y if we wanted to handle multiple options,
> but the Bison productions will work.
>
> There's nothing particularly stopping us from accepting
> "materialized" with a D in this syntax, instead of or in addition
> to "materialize"; though I hesitate to mention it for fear of
> another round of bikeshedding.
I think "materialize" is the right word since "materialized" would be
past tense.
> After further reflection I really don't like Andrew's suggestion
> that we not document the rule that multiply-referenced CTEs won't
> be inlined by default. That would be giving up the principle
> that WITH calculations are not done multiple times by default,
> and I draw the line at that. It's an often-useful behavior as
> well as one that's been documented from day one, so I do not accept
> the argument that we might someday override it on the basis of
> nothing but planner cost estimates.
Thinking of the history of documenting optimizer issues, I think we
should document when CTEs are inlined by default, because the user will
want to know when they should override the default behavior. When we
didn't document how PREPARED queries worked, we got many questions about
odd query performance until we finally documented it in 2016 in commit
fab9d1da4a213fab08fe2d263eedf2408bc4a27a. If we change the inlining
behavior later, we can update the docs.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-11 18:19 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 18:58 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 19:04 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 19:10 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-17 15:48 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-18 20:34 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-21 13:40 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-21 15:27 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 19:00 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-28 20:35 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 22:05 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-29 04:51 ` Re: Early WIP/PoC for inlining CTEs Michael Paquier <michael@paquier.xyz>
2019-01-29 19:52 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-02 19:01 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-06 09:00 ` Re: Early WIP/PoC for inlining CTEs Bruce Momjian <bruce@momjian.us>
@ 2019-02-14 15:25 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-02-15 01:15 ` Re: Early WIP/PoC for inlining CTEs Bruce Momjian <bruce@momjian.us>
0 siblings, 1 reply; 112+ messages in thread
From: Peter Eisentraut @ 2019-02-14 15:25 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Michael Paquier <michael@paquier.xyz>; Robert Haas <robertmhaas@gmail.com>; Andreas Karlsson <andreas@proxel.se>; Andres Freund <andres@anarazel.de>; Thomas Munro <thomas.munro@enterprisedb.com>; Andrew Gierth <andrew@tao11.riddles.org.uk>; David Fetter <david@fetter.org>; pgsql-hackers
On 06/02/2019 10:00, Bruce Momjian wrote:
> I think "materialize" is the right word since "materialized" would be
> past tense.
It's an adjective. The sentence is, "with foo as the materialized
$query, do the $main_query".
It's the same as "materialized view".
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-11 18:19 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 18:58 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 19:04 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 19:10 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-17 15:48 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-18 20:34 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-21 13:40 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-21 15:27 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 19:00 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-28 20:35 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 22:05 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-29 04:51 ` Re: Early WIP/PoC for inlining CTEs Michael Paquier <michael@paquier.xyz>
2019-01-29 19:52 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-02 19:01 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-02-06 09:00 ` Re: Early WIP/PoC for inlining CTEs Bruce Momjian <bruce@momjian.us>
2019-02-14 15:25 ` Re: Early WIP/PoC for inlining CTEs Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
@ 2019-02-15 01:15 ` Bruce Momjian <bruce@momjian.us>
0 siblings, 0 replies; 112+ messages in thread
From: Bruce Momjian @ 2019-02-15 01:15 UTC (permalink / raw)
To: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Michael Paquier <michael@paquier.xyz>; Robert Haas <robertmhaas@gmail.com>; Andreas Karlsson <andreas@proxel.se>; Andres Freund <andres@anarazel.de>; Thomas Munro <thomas.munro@enterprisedb.com>; Andrew Gierth <andrew@tao11.riddles.org.uk>; David Fetter <david@fetter.org>; pgsql-hackers
On Thu, Feb 14, 2019 at 04:25:27PM +0100, Peter Eisentraut wrote:
> On 06/02/2019 10:00, Bruce Momjian wrote:
> > I think "materialize" is the right word since "materialized" would be
> > past tense.
>
> It's an adjective. The sentence is, "with foo as the materialized
> $query, do the $main_query".
>
> It's the same as "materialized view".
Agreed, thanks.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-11 18:19 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 18:58 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 19:04 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 19:10 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-17 15:48 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-18 20:34 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-21 13:40 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-21 15:27 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 19:00 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-28 20:35 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 22:05 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
@ 2019-02-03 11:16 ` Vik Fearing <vik.fearing@2ndquadrant.com>
2 siblings, 0 replies; 112+ messages in thread
From: Vik Fearing @ 2019-02-03 11:16 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; +Cc: Robert Haas <robertmhaas@gmail.com>; Andreas Karlsson <andreas@proxel.se>; Andres Freund <andres@anarazel.de>; Thomas Munro <thomas.munro@enterprisedb.com>; Andrew Gierth <andrew@tao11.riddles.org.uk>; David Fetter <david@fetter.org>; pgsql-hackers
On 28/01/2019 23:05, Tom Lane wrote:
> Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:
>> On 28/01/2019 21:35, Tom Lane wrote:
>>> Conceivably we could make it work without the parens:
>>> ...
>
>> Or put it at the end?
>> WITH ctename AS ( query ) MATERIALIZED
>
> Yeah, I thought about that too, but it doesn't seem like an improvement.
> If the query is very long (which isn't unlikely) I think people would
> prefer to see the option(s) up front.
On the other hand, the end is where the other options go (that we
haven't implemented yet). See <search or cycle clause>.
--
Vik Fearing +33 6 46 75 15 36
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-11 18:19 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 18:58 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 19:04 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 19:10 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-17 15:48 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-18 20:34 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-21 13:40 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-21 15:27 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 19:00 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-28 20:35 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 22:05 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
@ 2019-02-03 15:39 ` Tom Lane <tgl@sss.pgh.pa.us>
2 siblings, 0 replies; 112+ messages in thread
From: Tom Lane @ 2019-02-03 15:39 UTC (permalink / raw)
To: Vik Fearing <vik.fearing@2ndquadrant.com>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Robert Haas <robertmhaas@gmail.com>; Andreas Karlsson <andreas@proxel.se>; Andres Freund <andres@anarazel.de>; Thomas Munro <thomas.munro@enterprisedb.com>; Andrew Gierth <andrew@tao11.riddles.org.uk>; David Fetter <david@fetter.org>; pgsql-hackers
Vik Fearing <vik.fearing@2ndquadrant.com> writes:
> On 28/01/2019 23:05, Tom Lane wrote:
>> Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:
>>> Or put it at the end?
>>> WITH ctename AS ( query ) MATERIALIZED
>> Yeah, I thought about that too, but it doesn't seem like an improvement.
>> If the query is very long (which isn't unlikely) I think people would
>> prefer to see the option(s) up front.
> On the other hand, the end is where the other options go (that we
> haven't implemented yet). See <search or cycle clause>.
Yeah, I noticed that too while working on the latest patch revision.
ISTM that's actually an argument for *not* putting PG-specific
syntax there. We'd increase the risk of conflicting with future spec
additions, assuming that they continue to add stuff at the end rather
than just after AS.
regards, tom lane
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-11 18:19 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 18:58 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 19:04 ` Re: Early WIP/PoC for inlining CTEs Andres Freund <andres@anarazel.de>
2019-01-11 19:10 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-17 15:48 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-18 20:34 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-21 13:40 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
@ 2019-01-22 05:13 ` Gavin Flower <GavinFlower@archidevsys.co.nz>
1 sibling, 0 replies; 112+ messages in thread
From: Gavin Flower @ 2019-01-22 05:13 UTC (permalink / raw)
To: pgsql-hackers@lists.postgresql.org
On 22/01/2019 02:40, Andreas Karlsson wrote:
> On 1/18/19 9:34 PM, Robert Haas wrote:
>> On Thu, Jan 17, 2019 at 10:48 AM Andreas Karlsson <andreas@proxel.se>
>> wrote:
>>> On 1/11/19 8:10 PM, Robert Haas wrote:
>>>> WITH cte_name [[NOT] MATERIALIZED] AS (query) main_query...
>>>
>>> Hm, when would one want "NOT MATERIALIZED"? I am not sure I see the
>>> usefulness of forcing inlining other than if we by default do not
>>> inline
>>> when a CTE is referenced multiple times.
>>
>> When the planner materializes it, but the performance of the resulting
>> plan therefore sucks, I suppose.
>>
>> I don't feel super-strongly about this, and Tom is right that there
>> may be cases where materialization is just not practical due to
>> implementation restrictions. But it's not crazy to imagine that
>> inlining a multiply-referenced CTE might create opportunities for
>> optimization at each of those places, perhaps not the same ones in
>> each case, whereas materializing it results in doing extra work.
>
> I see.
>
> I have a minor biksheddish question about the syntax.
>
> You proposed:
>
> WITH cte_name [[NOT] MATERIALIZED] AS (query) main_query
>
> While Andrew proposed:
>
> WITH cte_name AS [[NOT] MATERIALIZED] (query) main_query
>
> Do people have any preference between these two?
>
> Andreas
>
+1
For putting the 'AS' earlier, 2nd option, I think it reads better.
Cheers,
Gavin
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
@ 2019-01-11 18:49 ` David Fetter <david@fetter.org>
2019-01-11 18:58 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
1 sibling, 1 reply; 112+ messages in thread
From: David Fetter @ 2019-01-11 18:49 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Thomas Munro <thomas.munro@enterprisedb.com>; Andreas Karlsson <andreas@proxel.se>; Andrew Gierth <andrew@tao11.riddles.org.uk>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
On Fri, Jan 11, 2019 at 11:12:25AM -0500, Robert Haas wrote:
> > I know this is a thorny topic, but I have to say that I am uneasy
> > about the MATERIALIZED syntax. Here's how you write that in some
> > other RDBMS that loves hints:
> >
> > WITH foo AS (SELECT /*+ MATERIALIZE */ ...)
> >
> > I understood that it was a long standing project policy that we don't
> > want planner hints, but now we have a proposal to support one with a
> > top-level non-standard syntax. If we take this syntax, should we not
> > also accept MATERIALIZED in front of subselects?
> >
> > -1
>
> I think this is unduly negative. Do you want to also remove the
> IMMUTABLE, STABLE, and VOLATILE keywords from functions because those
> are hints to the planner as to how those things should be treated?
I don't see those as the same thing even slightly. Functions are
Turing complete, generally speaking, which means that unless we send
along those descriptors, we're asking the planner to solve the Halting
Problem.
> Should we remove CREATE INDEX -- which is, after all, a non-standard
> extension to SQL syntax -- because it presumes that the user is in a
> better position than we are to know which columns ought to be
> indexed?
Not yet. Maybe in a decade or two.
> OK, I know that's a bit of a straw man -- you're talking about hints
> within a query, not DDL. Still, I think our theory about not having
> hints is that we should have the optimizer try to figure it out
> instead of making the user specify the behavior that they want -- and
> I think sometimes that's setting the bar at an impossible level.
There is a worked example that's open source.
https://github.com/ossc-db/pg_hint_plan
Have we looked over it seriously for inclusion in PostgreSQL?
> It's not like this is a thing where we can get this right 90% of the
> time and with some more planner work we can drive it up to near 100%.
> We're going to be wrong a lot, even if we do expensive things like try
> planning it both ways and see which one comes out cheaper on cost, and
> we don't like driving up planner CPU consumption, either. So it seems
> to me that letting the user say what they want is a very pragmatic
> approach. Sometimes, like with things that have side effects, it's
> the only way to know that we're even delivering the right answer; and
> even when it's just an optimization problem, it's nice to give users a
> way of fixing our planning failures that is better than asking them to
> wait until we invent a way of improving the optimization decision in
> some future release - which we may never even do.
>
> I actually think that we should go "all in" here and allow the user to
> specify either that they want materialization or that they don't want
> materialization. If they specify neither, then we make some decision
> which we may change in a future release. If they do specify
> something, we do that.
When they're specifying it, are they specifying it globally, or
per WITH clause, or...?
Best,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778
Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
2019-01-11 18:49 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
@ 2019-01-11 18:58 ` Robert Haas <robertmhaas@gmail.com>
0 siblings, 0 replies; 112+ messages in thread
From: Robert Haas @ 2019-01-11 18:58 UTC (permalink / raw)
To: David Fetter <david@fetter.org>; +Cc: Thomas Munro <thomas.munro@enterprisedb.com>; Andreas Karlsson <andreas@proxel.se>; Andrew Gierth <andrew@tao11.riddles.org.uk>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
On Fri, Jan 11, 2019 at 1:49 PM David Fetter <david@fetter.org> wrote:
> I don't see those as the same thing even slightly. Functions are
> Turing complete, generally speaking, which means that unless we send
> along those descriptors, we're asking the planner to solve the Halting
> Problem.
So... your argument is that functions are Turing-complete, but the
queries which call those functions somehow aren't? Actually, there's
probably a decent argument that WITH RECURSIVE is Turing-complete even
without any fancy functions. See
https://wiki.postgresql.org/wiki/Turing_Machine_(with_recursive)
> > OK, I know that's a bit of a straw man -- you're talking about hints
> > within a query, not DDL. Still, I think our theory about not having
> > hints is that we should have the optimizer try to figure it out
> > instead of making the user specify the behavior that they want -- and
> > I think sometimes that's setting the bar at an impossible level.
>
> There is a worked example that's open source.
> https://github.com/ossc-db/pg_hint_plan
>
> Have we looked over it seriously for inclusion in PostgreSQL?
That really has very little to do with what's under discussion here,
unless you're proposing that the right strategy for determining
whether to materialize the CTE or not is to execute the query both
ways and then use that to construct the plan we use to execute the
query.
> When they're specifying it, are they specifying it globally, or
> per WITH clause, or...?
Per WITH clause. That's the proposal which is under discussion here,
not anything else. Did you read the thread?
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
@ 2019-01-22 03:47 ` Andreas Karlsson <andreas@proxel.se>
2019-01-25 22:22 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 1 reply; 112+ messages in thread
From: Andreas Karlsson @ 2019-01-22 03:47 UTC (permalink / raw)
To: Andrew Gierth <andrew@tao11.riddles.org.uk>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; David Fetter <david@fetter.org>; Thomas Munro <thomas.munro@enterprisedb.com>; pgsql-hackers
On 1/10/19 2:28 AM, Andreas Karlsson wrote:
> Here is a new version of the patch with added tests, improved comments,
> some minor code cleanup and most importantly slightly changed logic for
> when we should inline.
Add ctematerialized to the JumbleExpr() in pg_stat_statements on
suggestion from Andrew Gierth. I think that is the correct thing to do
since it can have a major impact on performance.
Andreas
Attachments:
[text/x-patch] inlining-ctes-v8.patch (29.6K, ../../29ce1707-0dae-ad33-8ab0-7d5226c7dfd3@proxel.se/2-inlining-ctes-v8.patch)
download | inline diff:
diff --git a/contrib/pg_stat_statements/pg_stat_statements.c b/contrib/pg_stat_statements/pg_stat_statements.c
index f177ebaa2c..6d456f6bce 100644
--- a/contrib/pg_stat_statements/pg_stat_statements.c
+++ b/contrib/pg_stat_statements/pg_stat_statements.c
@@ -2925,6 +2925,7 @@ JumbleExpr(pgssJumbleState *jstate, Node *node)
/* we store the string name because RTE_CTE RTEs need it */
APP_JUMB_STRING(cte->ctename);
+ APP_JUMB(cte->ctematerialized);
JumbleQuery(jstate, castNode(Query, cte->ctequery));
}
break;
diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out
index bb92d9d37a..8c26dd1f26 100644
--- a/contrib/postgres_fdw/expected/postgres_fdw.out
+++ b/contrib/postgres_fdw/expected/postgres_fdw.out
@@ -1888,7 +1888,7 @@ SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) ORDER BY t1.c3, t
-- join in CTE
EXPLAIN (VERBOSE, COSTS OFF)
-WITH t (c1_1, c1_3, c2_1) AS (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
+WITH t (c1_1, c1_3, c2_1) AS MATERIALIZED (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
QUERY PLAN
-------------------------------------------------------------------------------------------------------------------------------------
Limit
@@ -1905,7 +1905,7 @@ WITH t (c1_1, c1_3, c2_1) AS (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2
Output: t.c1_1, t.c2_1, t.c1_3
(12 rows)
-WITH t (c1_1, c1_3, c2_1) AS (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
+WITH t (c1_1, c1_3, c2_1) AS MATERIALIZED (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
c1_1 | c2_1
------+------
101 | 101
diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql
index f438165650..56602a164c 100644
--- a/contrib/postgres_fdw/sql/postgres_fdw.sql
+++ b/contrib/postgres_fdw/sql/postgres_fdw.sql
@@ -493,8 +493,8 @@ SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) ORDER BY t1.c3, t
SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) ORDER BY t1.c3, t1.c1 OFFSET 100 LIMIT 10 FOR SHARE;
-- join in CTE
EXPLAIN (VERBOSE, COSTS OFF)
-WITH t (c1_1, c1_3, c2_1) AS (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
-WITH t (c1_1, c1_3, c2_1) AS (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
+WITH t (c1_1, c1_3, c2_1) AS MATERIALIZED (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
+WITH t (c1_1, c1_3, c2_1) AS MATERIALIZED (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
-- ctid with whole-row reference
EXPLAIN (VERBOSE, COSTS OFF)
SELECT t1.ctid, t1, t2, t1.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) ORDER BY t1.c3, t1.c1 OFFSET 100 LIMIT 10;
diff --git a/doc/src/sgml/queries.sgml b/doc/src/sgml/queries.sgml
index 88bc189646..92f180c5cf 100644
--- a/doc/src/sgml/queries.sgml
+++ b/doc/src/sgml/queries.sgml
@@ -2199,6 +2199,8 @@ SELECT n FROM t LIMIT 100;
</para>
<para>
+ TODO: Update this for inlining and <literal>MATERIALIZED</literal>.
+
A useful property of <literal>WITH</literal> queries is that they are evaluated
only once per execution of the parent query, even if they are referred to
more than once by the parent query or sibling <literal>WITH</literal> queries.
diff --git a/doc/src/sgml/ref/select.sgml b/doc/src/sgml/ref/select.sgml
index 4db8142afa..92ede4fc6c 100644
--- a/doc/src/sgml/ref/select.sgml
+++ b/doc/src/sgml/ref/select.sgml
@@ -72,7 +72,7 @@ SELECT [ ALL | DISTINCT [ ON ( <replaceable class="parameter">expression</replac
<phrase>and <replaceable class="parameter">with_query</replaceable> is:</phrase>
- <replaceable class="parameter">with_query_name</replaceable> [ ( <replaceable class="parameter">column_name</replaceable> [, ...] ) ] AS ( <replaceable class="parameter">select</replaceable> | <replaceable class="parameter">values</replaceable> | <replaceable class="parameter">insert</replaceable> | <replaceable class="parameter">update</replaceable> | <replaceable class="parameter">delete</replaceable> )
+ <replaceable class="parameter">with_query_name</replaceable> [ ( <replaceable class="parameter">column_name</replaceable> [, ...] ) ] AS [ MATERIALIZED ] ( <replaceable class="parameter">select</replaceable> | <replaceable class="parameter">values</replaceable> | <replaceable class="parameter">insert</replaceable> | <replaceable class="parameter">update</replaceable> | <replaceable class="parameter">delete</replaceable> )
TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ]
</synopsis>
@@ -273,6 +273,8 @@ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ]
</para>
<para>
+ TODO: Update this for inlining and <literal>MATERIALIZED</literal>.
+
A key property of <literal>WITH</literal> queries is that they
are evaluated only once per execution of the primary query,
even if the primary query refers to them more than once.
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 3eb7e95d64..d5ecc0c389 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2539,6 +2539,7 @@ _copyCommonTableExpr(const CommonTableExpr *from)
COPY_NODE_FIELD(ctequery);
COPY_LOCATION_FIELD(location);
COPY_SCALAR_FIELD(cterecursive);
+ COPY_SCALAR_FIELD(ctematerialized);
COPY_SCALAR_FIELD(cterefcount);
COPY_NODE_FIELD(ctecolnames);
COPY_NODE_FIELD(ctecoltypes);
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index 5c4fa7d077..97cfe57328 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2794,6 +2794,7 @@ _equalCommonTableExpr(const CommonTableExpr *a, const CommonTableExpr *b)
COMPARE_NODE_FIELD(ctequery);
COMPARE_LOCATION_FIELD(location);
COMPARE_SCALAR_FIELD(cterecursive);
+ COMPARE_SCALAR_FIELD(ctematerialized);
COMPARE_SCALAR_FIELD(cterefcount);
COMPARE_NODE_FIELD(ctecolnames);
COMPARE_NODE_FIELD(ctecoltypes);
diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c
index 19b65f681d..03d085eda4 100644
--- a/src/backend/nodes/nodeFuncs.c
+++ b/src/backend/nodes/nodeFuncs.c
@@ -2314,8 +2314,7 @@ range_table_walker(List *rtable,
{
RangeTblEntry *rte = (RangeTblEntry *) lfirst(rt);
- /* For historical reasons, visiting RTEs is not the default */
- if (flags & QTW_EXAMINE_RTES)
+ if (flags & QTW_EXAMINE_RTES_BEFORE)
if (walker(rte, context))
return true;
@@ -2353,6 +2352,10 @@ range_table_walker(List *rtable,
break;
}
+ if (flags & QTW_EXAMINE_RTES_AFTER)
+ if (walker(rte, context))
+ return true;
+
if (walker(rte->securityQuals, context))
return true;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 0fde876c77..764bf36674 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2983,6 +2983,7 @@ _outCommonTableExpr(StringInfo str, const CommonTableExpr *node)
WRITE_NODE_FIELD(ctequery);
WRITE_LOCATION_FIELD(location);
WRITE_BOOL_FIELD(cterecursive);
+ WRITE_BOOL_FIELD(ctematerialized);
WRITE_INT_FIELD(cterefcount);
WRITE_NODE_FIELD(ctecolnames);
WRITE_NODE_FIELD(ctecoltypes);
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index ec6f2569ab..bd9086cb3f 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -421,6 +421,7 @@ _readCommonTableExpr(void)
READ_NODE_FIELD(ctequery);
READ_LOCATION_FIELD(location);
READ_BOOL_FIELD(cterecursive);
+ READ_BOOL_FIELD(ctematerialized);
READ_INT_FIELD(cterefcount);
READ_NODE_FIELD(ctecolnames);
READ_NODE_FIELD(ctecoltypes);
diff --git a/src/backend/optimizer/plan/setrefs.c b/src/backend/optimizer/plan/setrefs.c
index 5d363edab8..6bd3b2d1ac 100644
--- a/src/backend/optimizer/plan/setrefs.c
+++ b/src/backend/optimizer/plan/setrefs.c
@@ -340,7 +340,7 @@ flatten_unplanned_rtes(PlannerGlobal *glob, RangeTblEntry *rte)
(void) query_tree_walker(rte->subquery,
flatten_rtes_walker,
(void *) glob,
- QTW_EXAMINE_RTES);
+ QTW_EXAMINE_RTES_BEFORE);
}
static bool
@@ -363,7 +363,7 @@ flatten_rtes_walker(Node *node, PlannerGlobal *glob)
return query_tree_walker((Query *) node,
flatten_rtes_walker,
(void *) glob,
- QTW_EXAMINE_RTES);
+ QTW_EXAMINE_RTES_BEFORE);
}
return expression_tree_walker(node, flatten_rtes_walker,
(void *) glob);
diff --git a/src/backend/optimizer/plan/subselect.c b/src/backend/optimizer/plan/subselect.c
index 64272dd650..fe929dd144 100644
--- a/src/backend/optimizer/plan/subselect.c
+++ b/src/backend/optimizer/plan/subselect.c
@@ -57,6 +57,14 @@ typedef struct finalize_primnode_context
Bitmapset *paramids; /* Non-local PARAM_EXEC paramids found */
} finalize_primnode_context;
+typedef struct inline_cte_walker_context
+{
+ const char *ctename;
+ int levelsup;
+ int refcount;
+ Query *ctequery;
+ CommonTableExpr *cte;
+} inline_cte_walker_context;
static Node *build_subplan(PlannerInfo *root, Plan *plan, PlannerInfo *subroot,
List *plan_params,
@@ -800,14 +808,129 @@ hash_ok_operator(OpExpr *expr)
}
}
+/*
+ * inline_cte_walker: recursively walk the query and inline CTE
+ *
+ * While walking the expression tree with take care to examine the RTEs and
+ * potentially replace a CTE RTE with a subquery RTE after recursing into the
+ * RTE to avoid recursing into the newly inlined CTE.
+ */
+static bool
+inline_cte_walker(Node *node, inline_cte_walker_context *context)
+{
+ if (!node)
+ return false;
+
+ if (IsA(node, Query))
+ {
+ Query *query = castNode(Query, node);
+
+ context->levelsup++;
+
+ /*
+ * Visit the actual RTE nodes after their contents so we can modify
+ * the RTE node without having to visit the newly inlined contents.
+ */
+ query_tree_walker(query, inline_cte_walker, context,
+ QTW_EXAMINE_RTES_AFTER);
+
+ context->levelsup--;
+
+ return false;
+ }
+ else if (IsA(node, RangeTblEntry))
+ {
+ RangeTblEntry *rte = castNode(RangeTblEntry, node);
+
+ if (rte->rtekind == RTE_CTE &&
+ strcmp(rte->ctename, context->ctename) == 0 &&
+ rte->ctelevelsup == context->levelsup)
+ {
+ Query *newquery = copyObject(context->ctequery);
+
+ /* Preserve outer references, for example to other CTEs */
+ if (context->levelsup > 0)
+ IncrementVarSublevelsUp((Node *) newquery, context->levelsup, 1);
+
+ /*
+ * Replace the CTE reference in the range table with a subquery
+ *
+ * A FOR UPDATE clause is treated as extending into views and
+ * subqueries, but not into CTEs. We preserve this distinction
+ * by not trying to push rowmarks into the new subquery.
+ */
+ rte->rtekind = RTE_SUBQUERY;
+ rte->subquery = newquery;
+ rte->security_barrier = false;
+
+ /* Zero out CTE specific fields */
+ rte->ctename = NULL;
+ rte->ctelevelsup = 0;
+ rte->self_reference = false;
+ rte->coltypes = NIL;
+ rte->coltypmods = NIL;
+ rte->colcollations = NIL;
+
+ context->refcount--;
+ }
+
+ return false;
+ }
+
+ return expression_tree_walker(node, inline_cte_walker, context);
+}
+
+/*
+ * inline_cte: replace all refernces to CTE with subqueries
+ */
+static void
+inline_cte(PlannerInfo *root, CommonTableExpr *cte)
+{
+ struct inline_cte_walker_context context;
+ context.ctequery = castNode(Query, cte->ctequery);
+ context.ctename = cte->ctename;
+ context.refcount = cte->cterefcount;
+ context.levelsup = -1;
+ context.cte = cte;
+ inline_cte_walker((Node *) root->parse, &context);
+ /* we must replace all references */
+ Assert(context.refcount == 0);
+}
+
+static bool
+contain_row_marks_walker(Node *node, void *context)
+{
+ if (!node)
+ return false;
+
+ if (IsA(node, Query))
+ {
+ Query *query = castNode(Query, node);
+
+ if (query->rowMarks != NIL)
+ return true;
+
+ return query_tree_walker(query, contain_row_marks_walker, context, 0);
+ }
+ return expression_tree_walker(node, contain_row_marks_walker, context);
+}
+
+/*
+ * contain_row_marks: check the query or any subquery contains row marks
+ */
+static bool
+contain_row_marks(Node *node)
+{
+ return contain_row_marks_walker(node, NULL);
+}
/*
* SS_process_ctes: process a query's WITH list
*
- * We plan each interesting WITH item and convert it to an initplan.
- * A side effect is to fill in root->cte_plan_ids with a list that
- * parallels root->parse->cteList and provides the subplan ID for
- * each CTE's initplan.
+ * We plan each interesting WITH item and either convert it to an initplan, or
+ * if allowed inline it as a subquery. A side effect is to fill in
+ * root->cte_plan_ids with a list that parallels root->parse->cteList and
+ * provides the subplan ID for each CTE's initplan, or a dummy ID when inlined.
*/
void
SS_process_ctes(PlannerInfo *root)
@@ -839,6 +962,27 @@ SS_process_ctes(PlannerInfo *root)
}
/*
+ * We always inline the CTE as a subquery as long as it has no side
+ * effects, whch in this context means that it is a SELECT which does
+ * not call any volatile functions and does not have any rowmarks.
+ *
+ * The user may also have requested to not inline the CTE with the
+ * MATERIALIZED keyword.
+ */
+ if (!cte->ctematerialized &&
+ cmdType == CMD_SELECT &&
+ !cte->cterecursive &&
+ !contain_volatile_functions(cte->ctequery) &&
+ !contain_row_marks(cte->ctequery))
+ {
+ inline_cte(root, cte);
+
+ /* Make a dummy entry in cte_plan_ids */
+ root->cte_plan_ids = lappend_int(root->cte_plan_ids, -1);
+ continue;
+ }
+
+ /*
* Copy the source Query node. Probably not necessary, but let's keep
* this similar to make_subplan.
*/
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index d8a3c2d4cc..caac119299 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -518,7 +518,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
Bit ConstBit BitWithLength BitWithoutLength
%type <str> character
%type <str> extract_arg
-%type <boolean> opt_varying opt_timezone opt_no_inherit
+%type <boolean> opt_varying opt_timezone opt_no_inherit opt_materialized
%type <ival> Iconst SignedIconst
%type <str> Sconst comment_text notify_payload
@@ -11376,17 +11376,23 @@ cte_list:
| cte_list ',' common_table_expr { $$ = lappend($1, $3); }
;
-common_table_expr: name opt_name_list AS '(' PreparableStmt ')'
+common_table_expr: name opt_name_list AS opt_materialized '(' PreparableStmt ')'
{
CommonTableExpr *n = makeNode(CommonTableExpr);
n->ctename = $1;
n->aliascolnames = $2;
- n->ctequery = $5;
+ n->ctequery = $6;
+ n->ctematerialized = $4;
n->location = @1;
$$ = (Node *) n;
}
;
+opt_materialized:
+ MATERIALIZED { $$ = true; }
+ | /*EMPTY*/ { $$ = false; }
+ ;
+
opt_with_clause:
with_clause { $$ = $1; }
| /*EMPTY*/ { $$ = NULL; }
diff --git a/src/backend/rewrite/rewriteManip.c b/src/backend/rewrite/rewriteManip.c
index acbe669294..fd7a751c9a 100644
--- a/src/backend/rewrite/rewriteManip.c
+++ b/src/backend/rewrite/rewriteManip.c
@@ -761,7 +761,7 @@ IncrementVarSublevelsUp_walker(Node *node,
result = query_tree_walker((Query *) node,
IncrementVarSublevelsUp_walker,
(void *) context,
- QTW_EXAMINE_RTES);
+ QTW_EXAMINE_RTES_BEFORE);
context->min_sublevels_up--;
return result;
}
@@ -785,7 +785,7 @@ IncrementVarSublevelsUp(Node *node, int delta_sublevels_up,
query_or_expression_tree_walker(node,
IncrementVarSublevelsUp_walker,
(void *) &context,
- QTW_EXAMINE_RTES);
+ QTW_EXAMINE_RTES_BEFORE);
}
/*
@@ -804,7 +804,7 @@ IncrementVarSublevelsUp_rtable(List *rtable, int delta_sublevels_up,
range_table_walker(rtable,
IncrementVarSublevelsUp_walker,
(void *) &context,
- QTW_EXAMINE_RTES);
+ QTW_EXAMINE_RTES_BEFORE);
}
diff --git a/src/backend/utils/adt/ruleutils.c b/src/backend/utils/adt/ruleutils.c
index 302df16b4a..eb0010dd23 100644
--- a/src/backend/utils/adt/ruleutils.c
+++ b/src/backend/utils/adt/ruleutils.c
@@ -5166,7 +5166,10 @@ get_with_clause(Query *query, deparse_context *context)
}
appendStringInfoChar(buf, ')');
}
- appendStringInfoString(buf, " AS (");
+ appendStringInfoString(buf, " AS ");
+ if (cte->ctematerialized)
+ appendStringInfoString(buf, "MATERIALIZED ");
+ appendStringInfoChar(buf, '(');
if (PRETTY_INDENT(context))
appendContextKeyword(context, "", 0, 0, 0);
get_query_def((Query *) cte->ctequery, buf, context->namespaces, NULL,
diff --git a/src/include/nodes/nodeFuncs.h b/src/include/nodes/nodeFuncs.h
index 7739600db2..2774f15cc8 100644
--- a/src/include/nodes/nodeFuncs.h
+++ b/src/include/nodes/nodeFuncs.h
@@ -22,8 +22,10 @@
#define QTW_IGNORE_RC_SUBQUERIES 0x03 /* both of above */
#define QTW_IGNORE_JOINALIASES 0x04 /* JOIN alias var lists */
#define QTW_IGNORE_RANGE_TABLE 0x08 /* skip rangetable entirely */
-#define QTW_EXAMINE_RTES 0x10 /* examine RTEs */
+#define QTW_EXAMINE_RTES_BEFORE 0x10 /* examine RTE nodes before their contents */
#define QTW_DONT_COPY_QUERY 0x20 /* do not copy top Query */
+#define QTW_EXAMINE_RTES_AFTER 0x40 /* examine RTE nodes after their contents */
+
/* callback function for check_functions_in_node */
typedef bool (*check_function_callback) (Oid func_id, void *context);
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index addc2c2ec7..e7e945057c 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -1409,6 +1409,7 @@ typedef struct CommonTableExpr
int location; /* token location, or -1 if unknown */
/* These fields are set during parse analysis: */
bool cterecursive; /* is this CTE actually recursive? */
+ bool ctematerialized; /* is this an optimization fence? */
int cterefcount; /* number of RTEs referencing this CTE
* (excluding internal self-references) */
List *ctecolnames; /* list of output column names */
diff --git a/src/test/regress/expected/rowsecurity.out b/src/test/regress/expected/rowsecurity.out
index 1d12b01068..1a0932d863 100644
--- a/src/test/regress/expected/rowsecurity.out
+++ b/src/test/regress/expected/rowsecurity.out
@@ -2179,7 +2179,7 @@ EXPLAIN (COSTS OFF) EXECUTE plancache_test;
Filter: (((a % 2) = 0) AND f_leak(b))
(2 rows)
-PREPARE plancache_test2 AS WITH q AS (SELECT * FROM z1 WHERE f_leak(b)) SELECT * FROM q,z2;
+PREPARE plancache_test2 AS WITH q AS MATERIALIZED (SELECT * FROM z1 WHERE f_leak(b)) SELECT * FROM q,z2;
EXPLAIN (COSTS OFF) EXECUTE plancache_test2;
QUERY PLAN
-------------------------------------------------
@@ -2192,7 +2192,7 @@ EXPLAIN (COSTS OFF) EXECUTE plancache_test2;
-> Seq Scan on z2
(7 rows)
-PREPARE plancache_test3 AS WITH q AS (SELECT * FROM z2) SELECT * FROM q,z1 WHERE f_leak(z1.b);
+PREPARE plancache_test3 AS WITH q AS MATERIALIZED (SELECT * FROM z2) SELECT * FROM q,z1 WHERE f_leak(z1.b);
EXPLAIN (COSTS OFF) EXECUTE plancache_test3;
QUERY PLAN
-----------------------------------------------------
@@ -2853,7 +2853,7 @@ NOTICE: f_leak => 98f13708210194c475687be6106a3b84
20 | 98f13708210194c475687be6106a3b84
(11 rows)
-EXPLAIN (COSTS OFF) WITH cte1 AS (SELECT * FROM t1 WHERE f_leak(b)) SELECT * FROM cte1;
+EXPLAIN (COSTS OFF) WITH cte1 AS MATERIALIZED (SELECT * FROM t1 WHERE f_leak(b)) SELECT * FROM cte1;
QUERY PLAN
-------------------------------------------------
CTE Scan on cte1
diff --git a/src/test/regress/expected/rowtypes.out b/src/test/regress/expected/rowtypes.out
index d6a1a3331e..14757b40f6 100644
--- a/src/test/regress/expected/rowtypes.out
+++ b/src/test/regress/expected/rowtypes.out
@@ -1037,7 +1037,7 @@ from (values (1,row(1,2)), (1,row(null,null)), (1,null),
(6 rows)
explain (verbose, costs off)
-with r(a,b) as
+with r(a,b) as materialized
(values (1,row(1,2)), (1,row(null,null)), (1,null),
(null,row(1,2)), (null,row(null,null)), (null,null) )
select r, r is null as isnull, r is not null as isnotnull from r;
@@ -1050,7 +1050,7 @@ select r, r is null as isnull, r is not null as isnotnull from r;
Output: "*VALUES*".column1, "*VALUES*".column2
(5 rows)
-with r(a,b) as
+with r(a,b) as materialized
(values (1,row(1,2)), (1,row(null,null)), (1,null),
(null,row(1,2)), (null,row(null,null)), (null,null) )
select r, r is null as isnull, r is not null as isnotnull from r;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index e384cd2279..32dfab94b2 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -3108,7 +3108,7 @@ explain (costs off) INSERT INTO hats VALUES ('h8', 'forbidden') RETURNING *;
(5 rows)
-- ensure upserting into a rule, with a CTE (different offsets!) works
-WITH data(hat_name, hat_color) AS (
+WITH data(hat_name, hat_color) AS MATERIALIZED (
VALUES ('h8', 'green'),
('h9', 'blue'),
('h7', 'forbidden')
@@ -3122,7 +3122,7 @@ RETURNING *;
h9 | blue
(2 rows)
-EXPLAIN (costs off) WITH data(hat_name, hat_color) AS (
+EXPLAIN (costs off) WITH data(hat_name, hat_color) AS MATERIALIZED (
VALUES ('h8', 'green'),
('h9', 'blue'),
('h7', 'forbidden')
diff --git a/src/test/regress/expected/subselect.out b/src/test/regress/expected/subselect.out
index 588d069589..d52dfe022a 100644
--- a/src/test/regress/expected/subselect.out
+++ b/src/test/regress/expected/subselect.out
@@ -1154,3 +1154,92 @@ fetch backward all in c1;
(2 rows)
commit;
+-- Ensure that we inline the currect CTE when there multiple CTEs with the same name
+explain (verbose, costs off)
+with x as (select 1 as y)
+select * from (with x as (select 2 as y) select * from x) ss;
+ QUERY PLAN
+-------------
+ Result
+ Output: 2
+(2 rows)
+
+-- Ensure that we do not inline when there are side effects
+-- Basic subquery which can be inlined
+explain (verbose, costs off)
+with x as (select * from (select f1 from subselect_tbl) ss)
+select * from x where f1 = 1;
+ QUERY PLAN
+----------------------------------
+ Seq Scan on public.subselect_tbl
+ Output: subselect_tbl.f1
+ Filter: (subselect_tbl.f1 = 1)
+(3 rows)
+
+-- Stable functions are safe to inline
+explain (verbose, costs off)
+with x as (select * from (select f1, now() from subselect_tbl) ss)
+select * from x where f1 = 1;
+ QUERY PLAN
+-----------------------------------
+ Seq Scan on public.subselect_tbl
+ Output: subselect_tbl.f1, now()
+ Filter: (subselect_tbl.f1 = 1)
+(3 rows)
+
+-- Volatile functions prevent inlining
+explain (verbose, costs off)
+with x as (select * from (select f1, random() from subselect_tbl) ss)
+select * from x where f1 = 1;
+ QUERY PLAN
+----------------------------------------------
+ CTE Scan on x
+ Output: x.f1, x.random
+ Filter: (x.f1 = 1)
+ CTE x
+ -> Seq Scan on public.subselect_tbl
+ Output: subselect_tbl.f1, random()
+(6 rows)
+
+-- Row marks prevent inlining
+explain (verbose, costs off)
+with x as (select * from (select f1 from subselect_tbl for update) ss)
+select * from x where f1 = 1;
+ QUERY PLAN
+--------------------------------------------------------------------
+ CTE Scan on x
+ Output: x.f1
+ Filter: (x.f1 = 1)
+ CTE x
+ -> Subquery Scan on ss
+ Output: ss.f1
+ -> LockRows
+ Output: subselect_tbl.f1, subselect_tbl.ctid
+ -> Seq Scan on public.subselect_tbl
+ Output: subselect_tbl.f1, subselect_tbl.ctid
+(10 rows)
+
+-- Explicitly request materialization
+explain (verbose, costs off)
+with x as materialized (select * from (select f1 from subselect_tbl) ss)
+select * from x where f1 = 1;
+ QUERY PLAN
+------------------------------------------
+ CTE Scan on x
+ Output: x.f1
+ Filter: (x.f1 = 1)
+ CTE x
+ -> Seq Scan on public.subselect_tbl
+ Output: subselect_tbl.f1
+(6 rows)
+
+-- Row marks are not pushed into CTEs
+explain (verbose, costs off)
+with x as (select * from subselect_tbl)
+select * from x for update;
+ QUERY PLAN
+----------------------------------------------------------------
+ Seq Scan on public.subselect_tbl
+ Output: subselect_tbl.f1, subselect_tbl.f2, subselect_tbl.f3
+(2 rows)
+
diff --git a/src/test/regress/sql/rowsecurity.sql b/src/test/regress/sql/rowsecurity.sql
index 38e9b38bc4..4c9e846fab 100644
--- a/src/test/regress/sql/rowsecurity.sql
+++ b/src/test/regress/sql/rowsecurity.sql
@@ -840,10 +840,10 @@ EXPLAIN (COSTS OFF) SELECT * FROM z1 WHERE f_leak(b);
PREPARE plancache_test AS SELECT * FROM z1 WHERE f_leak(b);
EXPLAIN (COSTS OFF) EXECUTE plancache_test;
-PREPARE plancache_test2 AS WITH q AS (SELECT * FROM z1 WHERE f_leak(b)) SELECT * FROM q,z2;
+PREPARE plancache_test2 AS WITH q AS MATERIALIZED (SELECT * FROM z1 WHERE f_leak(b)) SELECT * FROM q,z2;
EXPLAIN (COSTS OFF) EXECUTE plancache_test2;
-PREPARE plancache_test3 AS WITH q AS (SELECT * FROM z2) SELECT * FROM q,z1 WHERE f_leak(z1.b);
+PREPARE plancache_test3 AS WITH q AS MATERIALIZED (SELECT * FROM z2) SELECT * FROM q,z1 WHERE f_leak(z1.b);
EXPLAIN (COSTS OFF) EXECUTE plancache_test3;
SET ROLE regress_rls_group1;
@@ -1072,7 +1072,7 @@ INSERT INTO t1 (SELECT x, md5(x::text) FROM generate_series(0,20) x);
SET SESSION AUTHORIZATION regress_rls_bob;
WITH cte1 AS (SELECT * FROM t1 WHERE f_leak(b)) SELECT * FROM cte1;
-EXPLAIN (COSTS OFF) WITH cte1 AS (SELECT * FROM t1 WHERE f_leak(b)) SELECT * FROM cte1;
+EXPLAIN (COSTS OFF) WITH cte1 AS MATERIALIZED (SELECT * FROM t1 WHERE f_leak(b)) SELECT * FROM cte1;
WITH cte1 AS (UPDATE t1 SET a = a + 1 RETURNING *) SELECT * FROM cte1; --fail
WITH cte1 AS (UPDATE t1 SET a = a RETURNING *) SELECT * FROM cte1; --ok
diff --git a/src/test/regress/sql/rowtypes.sql b/src/test/regress/sql/rowtypes.sql
index e6d389805c..b3b07746c3 100644
--- a/src/test/regress/sql/rowtypes.sql
+++ b/src/test/regress/sql/rowtypes.sql
@@ -428,12 +428,12 @@ from (values (1,row(1,2)), (1,row(null,null)), (1,null),
(null,row(1,2)), (null,row(null,null)), (null,null) ) r(a,b);
explain (verbose, costs off)
-with r(a,b) as
+with r(a,b) as materialized
(values (1,row(1,2)), (1,row(null,null)), (1,null),
(null,row(1,2)), (null,row(null,null)), (null,null) )
select r, r is null as isnull, r is not null as isnotnull from r;
-with r(a,b) as
+with r(a,b) as materialized
(values (1,row(1,2)), (1,row(null,null)), (1,null),
(null,row(1,2)), (null,row(null,null)), (null,null) )
select r, r is null as isnull, r is not null as isnotnull from r;
diff --git a/src/test/regress/sql/rules.sql b/src/test/regress/sql/rules.sql
index f4ee30ec8f..82750b8e6f 100644
--- a/src/test/regress/sql/rules.sql
+++ b/src/test/regress/sql/rules.sql
@@ -1132,7 +1132,7 @@ SELECT tablename, rulename, definition FROM pg_rules
explain (costs off) INSERT INTO hats VALUES ('h8', 'forbidden') RETURNING *;
-- ensure upserting into a rule, with a CTE (different offsets!) works
-WITH data(hat_name, hat_color) AS (
+WITH data(hat_name, hat_color) AS MATERIALIZED (
VALUES ('h8', 'green'),
('h9', 'blue'),
('h7', 'forbidden')
@@ -1140,7 +1140,7 @@ WITH data(hat_name, hat_color) AS (
INSERT INTO hats
SELECT * FROM data
RETURNING *;
-EXPLAIN (costs off) WITH data(hat_name, hat_color) AS (
+EXPLAIN (costs off) WITH data(hat_name, hat_color) AS MATERIALIZED (
VALUES ('h8', 'green'),
('h9', 'blue'),
('h7', 'forbidden')
diff --git a/src/test/regress/sql/subselect.sql b/src/test/regress/sql/subselect.sql
index 843f511b3d..bdbba3838f 100644
--- a/src/test/regress/sql/subselect.sql
+++ b/src/test/regress/sql/subselect.sql
@@ -625,3 +625,42 @@ move forward all in c1;
fetch backward all in c1;
commit;
+
+-- Ensure that we inline the currect CTE when there multiple CTEs with the same name
+
+explain (verbose, costs off)
+with x as (select 1 as y)
+select * from (with x as (select 2 as y) select * from x) ss;
+
+-- Ensure that we do not inline when there are side effects
+
+-- Basic subquery which can be inlined
+explain (verbose, costs off)
+with x as (select * from (select f1 from subselect_tbl) ss)
+select * from x where f1 = 1;
+
+-- Stable functions are safe to inline
+explain (verbose, costs off)
+with x as (select * from (select f1, now() from subselect_tbl) ss)
+select * from x where f1 = 1;
+
+-- Volatile functions prevent inlining
+explain (verbose, costs off)
+with x as (select * from (select f1, random() from subselect_tbl) ss)
+select * from x where f1 = 1;
+
+-- Row marks prevent inlining
+explain (verbose, costs off)
+with x as (select * from (select f1 from subselect_tbl for update) ss)
+select * from x where f1 = 1;
+
+-- Explicitly request materialization
+explain (verbose, costs off)
+with x as materialized (select * from (select f1 from subselect_tbl) ss)
+select * from x where f1 = 1;
+
+-- Row marks are not pushed into CTEs
+
+explain (verbose, costs off)
+with x as (select * from subselect_tbl)
+select * from x for update;
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-22 03:47 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
@ 2019-01-25 22:22 ` Tom Lane <tgl@sss.pgh.pa.us>
2019-01-26 22:55 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-27 00:15 ` Re: Early WIP/PoC for inlining CTEs Marko Tiikkaja <marko@joh.to>
0 siblings, 2 replies; 112+ messages in thread
From: Tom Lane @ 2019-01-25 22:22 UTC (permalink / raw)
To: Andreas Karlsson <andreas@proxel.se>; +Cc: Andrew Gierth <andrew@tao11.riddles.org.uk>; David Fetter <david@fetter.org>; Thomas Munro <thomas.munro@enterprisedb.com>; pgsql-hackers
Andreas Karlsson <andreas@proxel.se> writes:
> [ inlining-ctes-v8.patch ]
I went ahead and pushed the stuff about QTW_EXAMINE_RTES_BEFORE/_AFTER,
because that seems like an independent change with other possible uses.
Attached is an updated version of the rest of the patch, with mostly
cosmetic changes. I've not touched the documentation, but I think this
is otherwise committable if we are satisfied with the semantics.
However ... after thinking about it more, I'm not really satisfied
with that. In particular I don't like the fact that by default this
will inline regardless of the number of references to the CTE. I doubt
that inlining when there are multiple references is so likely to be a
win as to justify it being the default, especially given that it flies
in the face of what our documentation has said for as long as we've
had CTEs.
Therefore, I'm reversing my previous opinion that we should not have
an explicit NOT MATERIALIZED option. I think we should add that, and
the behavior ought to be:
* No option given: inline if there's exactly one reference.
* With MATERIALIZED: never inline.
* With NOT MATERIALIZED: inline regardless of the number of references.
(Obviously, we should not inline if there's RECURSIVE or the CTE
potentially has side-effects, regardless of the user option;
I don't think those cases are up for debate.)
I haven't done anything about that here, but the changes would be pretty
minor.
regards, tom lane
Attachments:
[text/x-diff] inlining-ctes-v9.patch (30.0K, ../../11140.1548454956@sss.pgh.pa.us/2-inlining-ctes-v9.patch)
download | inline diff:
diff --git a/contrib/pg_stat_statements/pg_stat_statements.c b/contrib/pg_stat_statements/pg_stat_statements.c
index f177eba..6d456f6 100644
--- a/contrib/pg_stat_statements/pg_stat_statements.c
+++ b/contrib/pg_stat_statements/pg_stat_statements.c
@@ -2925,6 +2925,7 @@ JumbleExpr(pgssJumbleState *jstate, Node *node)
/* we store the string name because RTE_CTE RTEs need it */
APP_JUMB_STRING(cte->ctename);
+ APP_JUMB(cte->ctematerialized);
JumbleQuery(jstate, castNode(Query, cte->ctequery));
}
break;
diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out
index bb92d9d..8c26dd1 100644
--- a/contrib/postgres_fdw/expected/postgres_fdw.out
+++ b/contrib/postgres_fdw/expected/postgres_fdw.out
@@ -1888,7 +1888,7 @@ SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) ORDER BY t1.c3, t
-- join in CTE
EXPLAIN (VERBOSE, COSTS OFF)
-WITH t (c1_1, c1_3, c2_1) AS (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
+WITH t (c1_1, c1_3, c2_1) AS MATERIALIZED (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
QUERY PLAN
-------------------------------------------------------------------------------------------------------------------------------------
Limit
@@ -1905,7 +1905,7 @@ WITH t (c1_1, c1_3, c2_1) AS (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2
Output: t.c1_1, t.c2_1, t.c1_3
(12 rows)
-WITH t (c1_1, c1_3, c2_1) AS (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
+WITH t (c1_1, c1_3, c2_1) AS MATERIALIZED (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
c1_1 | c2_1
------+------
101 | 101
diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql
index f438165..56602a1 100644
--- a/contrib/postgres_fdw/sql/postgres_fdw.sql
+++ b/contrib/postgres_fdw/sql/postgres_fdw.sql
@@ -493,8 +493,8 @@ SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) ORDER BY t1.c3, t
SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) ORDER BY t1.c3, t1.c1 OFFSET 100 LIMIT 10 FOR SHARE;
-- join in CTE
EXPLAIN (VERBOSE, COSTS OFF)
-WITH t (c1_1, c1_3, c2_1) AS (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
-WITH t (c1_1, c1_3, c2_1) AS (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
+WITH t (c1_1, c1_3, c2_1) AS MATERIALIZED (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
+WITH t (c1_1, c1_3, c2_1) AS MATERIALIZED (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
-- ctid with whole-row reference
EXPLAIN (VERBOSE, COSTS OFF)
SELECT t1.ctid, t1, t2, t1.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) ORDER BY t1.c3, t1.c1 OFFSET 100 LIMIT 10;
diff --git a/doc/src/sgml/queries.sgml b/doc/src/sgml/queries.sgml
index 88bc189..92f180c 100644
--- a/doc/src/sgml/queries.sgml
+++ b/doc/src/sgml/queries.sgml
@@ -2199,6 +2199,8 @@ SELECT n FROM t LIMIT 100;
</para>
<para>
+ TODO: Update this for inlining and <literal>MATERIALIZED</literal>.
+
A useful property of <literal>WITH</literal> queries is that they are evaluated
only once per execution of the parent query, even if they are referred to
more than once by the parent query or sibling <literal>WITH</literal> queries.
diff --git a/doc/src/sgml/ref/select.sgml b/doc/src/sgml/ref/select.sgml
index 4db8142..92ede4f 100644
--- a/doc/src/sgml/ref/select.sgml
+++ b/doc/src/sgml/ref/select.sgml
@@ -72,7 +72,7 @@ SELECT [ ALL | DISTINCT [ ON ( <replaceable class="parameter">expression</replac
<phrase>and <replaceable class="parameter">with_query</replaceable> is:</phrase>
- <replaceable class="parameter">with_query_name</replaceable> [ ( <replaceable class="parameter">column_name</replaceable> [, ...] ) ] AS ( <replaceable class="parameter">select</replaceable> | <replaceable class="parameter">values</replaceable> | <replaceable class="parameter">insert</replaceable> | <replaceable class="parameter">update</replaceable> | <replaceable class="parameter">delete</replaceable> )
+ <replaceable class="parameter">with_query_name</replaceable> [ ( <replaceable class="parameter">column_name</replaceable> [, ...] ) ] AS [ MATERIALIZED ] ( <replaceable class="parameter">select</replaceable> | <replaceable class="parameter">values</replaceable> | <replaceable class="parameter">insert</replaceable> | <replaceable class="parameter">update</replaceable> | <replaceable class="parameter">delete</replaceable> )
TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ]
</synopsis>
@@ -273,6 +273,8 @@ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ]
</para>
<para>
+ TODO: Update this for inlining and <literal>MATERIALIZED</literal>.
+
A key property of <literal>WITH</literal> queries is that they
are evaluated only once per execution of the primary query,
even if the primary query refers to them more than once.
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 3eb7e95..3b6e206 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2536,6 +2536,7 @@ _copyCommonTableExpr(const CommonTableExpr *from)
COPY_STRING_FIELD(ctename);
COPY_NODE_FIELD(aliascolnames);
+ COPY_SCALAR_FIELD(ctematerialized);
COPY_NODE_FIELD(ctequery);
COPY_LOCATION_FIELD(location);
COPY_SCALAR_FIELD(cterecursive);
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index 5c4fa7d..e7b0fae 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2791,6 +2791,7 @@ _equalCommonTableExpr(const CommonTableExpr *a, const CommonTableExpr *b)
{
COMPARE_STRING_FIELD(ctename);
COMPARE_NODE_FIELD(aliascolnames);
+ COMPARE_SCALAR_FIELD(ctematerialized);
COMPARE_NODE_FIELD(ctequery);
COMPARE_LOCATION_FIELD(location);
COMPARE_SCALAR_FIELD(cterecursive);
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 0fde876..b8467cd 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2980,6 +2980,7 @@ _outCommonTableExpr(StringInfo str, const CommonTableExpr *node)
WRITE_STRING_FIELD(ctename);
WRITE_NODE_FIELD(aliascolnames);
+ WRITE_BOOL_FIELD(ctematerialized);
WRITE_NODE_FIELD(ctequery);
WRITE_LOCATION_FIELD(location);
WRITE_BOOL_FIELD(cterecursive);
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index ec6f256..c653ee3 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -418,6 +418,7 @@ _readCommonTableExpr(void)
READ_STRING_FIELD(ctename);
READ_NODE_FIELD(aliascolnames);
+ READ_BOOL_FIELD(ctematerialized);
READ_NODE_FIELD(ctequery);
READ_LOCATION_FIELD(location);
READ_BOOL_FIELD(cterecursive);
diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c
index 4465f00..67f276a 100644
--- a/src/backend/optimizer/plan/planner.c
+++ b/src/backend/optimizer/plan/planner.c
@@ -645,8 +645,8 @@ subquery_planner(PlannerGlobal *glob, Query *parse,
root->partColsUpdated = false;
/*
- * If there is a WITH list, process each WITH query and build an initplan
- * SubPlan structure for it.
+ * If there is a WITH list, process each WITH query and either convert it
+ * to RTE_SUBQUERY RTE(s) or build an initplan SubPlan structure for it.
*/
if (parse->cteList)
SS_process_ctes(root);
diff --git a/src/backend/optimizer/plan/subselect.c b/src/backend/optimizer/plan/subselect.c
index 64272dd..8dd6a20 100644
--- a/src/backend/optimizer/plan/subselect.c
+++ b/src/backend/optimizer/plan/subselect.c
@@ -57,6 +57,14 @@ typedef struct finalize_primnode_context
Bitmapset *paramids; /* Non-local PARAM_EXEC paramids found */
} finalize_primnode_context;
+typedef struct inline_cte_walker_context
+{
+ const char *ctename; /* name and relative level of target CTE */
+ int levelsup;
+ int refcount; /* number of remaining references */
+ Query *ctequery; /* query to substitute */
+} inline_cte_walker_context;
+
static Node *build_subplan(PlannerInfo *root, Plan *plan, PlannerInfo *subroot,
List *plan_params,
@@ -75,6 +83,10 @@ static Node *convert_testexpr_mutator(Node *node,
static bool subplan_is_hashable(Plan *plan);
static bool testexpr_is_hashable(Node *testexpr);
static bool hash_ok_operator(OpExpr *expr);
+static bool contain_dml(Node *node);
+static bool contain_dml_walker(Node *node, void *context);
+static void inline_cte(PlannerInfo *root, CommonTableExpr *cte);
+static bool inline_cte_walker(Node *node, inline_cte_walker_context *context);
static bool simplify_EXISTS_query(PlannerInfo *root, Query *query);
static Query *convert_EXISTS_to_ANY(PlannerInfo *root, Query *subselect,
Node **testexpr, List **paramIds);
@@ -804,10 +816,13 @@ hash_ok_operator(OpExpr *expr)
/*
* SS_process_ctes: process a query's WITH list
*
- * We plan each interesting WITH item and convert it to an initplan.
+ * Consider each CTE in the WITH list and either ignore it (if it's an
+ * unreferenced SELECT), "inline" it to create a regular sub-SELECT-in-FROM,
+ * or convert it to an initplan.
+ *
* A side effect is to fill in root->cte_plan_ids with a list that
* parallels root->parse->cteList and provides the subplan ID for
- * each CTE's initplan.
+ * each CTE's initplan, or a dummy ID (-1) if we didn't make an initplan.
*/
void
SS_process_ctes(PlannerInfo *root)
@@ -839,6 +854,39 @@ SS_process_ctes(PlannerInfo *root)
}
/*
+ * Consider inlining the CTE (creating RTE_SUBQUERY RTE(s)) instead of
+ * implementing it as a separately-planned CTE.
+ *
+ * We cannot inline if any of these conditions hold:
+ *
+ * 1. The user said not to (the MATERIALIZED option).
+ *
+ * 2. The CTE is recursive.
+ *
+ * 3. The CTE has side-effects; this includes either not being a plain
+ * SELECT, or containing volatile functions. Inlining might change
+ * the side-effects, which is not good.
+ *
+ * Otherwise, we always inline. This could be a net loss if the CTE
+ * is multiply referenced, but not necessarily so: the ability to
+ * absorb restrictions from the outer query level could outweigh the
+ * need for duplicate computations. We do not have nearly enough
+ * information at this point to tell if that's true, so we make the
+ * user say MATERIALIZED if it's not.
+ */
+ if (!cte->ctematerialized &&
+ !cte->cterecursive &&
+ cmdType == CMD_SELECT &&
+ !contain_dml(cte->ctequery) &&
+ !contain_volatile_functions(cte->ctequery))
+ {
+ inline_cte(root, cte);
+ /* Make a dummy entry in cte_plan_ids */
+ root->cte_plan_ids = lappend_int(root->cte_plan_ids, -1);
+ continue;
+ }
+
+ /*
* Copy the source Query node. Probably not necessary, but let's keep
* this similar to make_subplan.
*/
@@ -935,6 +983,127 @@ SS_process_ctes(PlannerInfo *root)
}
/*
+ * contain_dml: is any subquery not a plain SELECT?
+ *
+ * We reject SELECT FOR UPDATE/SHARE as well as INSERT etc.
+ */
+static bool
+contain_dml(Node *node)
+{
+ return contain_dml_walker(node, NULL);
+}
+
+static bool
+contain_dml_walker(Node *node, void *context)
+{
+ if (node == NULL)
+ return false;
+ if (IsA(node, Query))
+ {
+ Query *query = (Query *) node;
+
+ if (query->commandType != CMD_SELECT ||
+ query->rowMarks != NIL)
+ return true;
+
+ return query_tree_walker(query, contain_dml_walker, context, 0);
+ }
+ return expression_tree_walker(node, contain_dml_walker, context);
+}
+
+/*
+ * inline_cte: convert RTE_CTE references to given CTE into RTE_SUBQUERYs
+ */
+static void
+inline_cte(PlannerInfo *root, CommonTableExpr *cte)
+{
+ struct inline_cte_walker_context context;
+
+ context.ctename = cte->ctename;
+ /* Start at levelsup = -1 because we'll immediately increment it */
+ context.levelsup = -1;
+ context.refcount = cte->cterefcount;
+ context.ctequery = castNode(Query, cte->ctequery);
+
+ (void) inline_cte_walker((Node *) root->parse, &context);
+
+ /* Assert we replaced all references */
+ Assert(context.refcount == 0);
+}
+
+static bool
+inline_cte_walker(Node *node, inline_cte_walker_context *context)
+{
+ if (node == NULL)
+ return false;
+ if (IsA(node, Query))
+ {
+ Query *query = (Query *) node;
+
+ context->levelsup++;
+
+ /*
+ * Visit the query's RTE nodes after their contents; otherwise
+ * query_tree_walker would descend into the newly inlined CTE query,
+ * which we don't want.
+ */
+ (void) query_tree_walker(query, inline_cte_walker, context,
+ QTW_EXAMINE_RTES_AFTER);
+
+ context->levelsup--;
+
+ return false;
+ }
+ else if (IsA(node, RangeTblEntry))
+ {
+ RangeTblEntry *rte = (RangeTblEntry *) node;
+
+ if (rte->rtekind == RTE_CTE &&
+ strcmp(rte->ctename, context->ctename) == 0 &&
+ rte->ctelevelsup == context->levelsup)
+ {
+ /*
+ * Found a reference to replace. Generate a copy of the CTE query
+ * with appropriate level adjustment for outer references (e.g.,
+ * to other CTEs).
+ */
+ Query *newquery = copyObject(context->ctequery);
+
+ if (context->levelsup > 0)
+ IncrementVarSublevelsUp((Node *) newquery, context->levelsup, 1);
+
+ /*
+ * Convert the RTE_CTE RTE into a RTE_SUBQUERY.
+ *
+ * Historically, a FOR UPDATE clause has been treated as extending
+ * into views and subqueries, but not into CTEs. We preserve this
+ * distinction by not trying to push rowmarks into the new
+ * subquery.
+ */
+ rte->rtekind = RTE_SUBQUERY;
+ rte->subquery = newquery;
+ rte->security_barrier = false;
+
+ /* Zero out CTE-specific fields */
+ rte->ctename = NULL;
+ rte->ctelevelsup = 0;
+ rte->self_reference = false;
+ rte->coltypes = NIL;
+ rte->coltypmods = NIL;
+ rte->colcollations = NIL;
+
+ /* Count the number of replacements we've done */
+ context->refcount--;
+ }
+
+ return false;
+ }
+
+ return expression_tree_walker(node, inline_cte_walker, context);
+}
+
+
+/*
* convert_ANY_sublink_to_join: try to convert an ANY SubLink to a join
*
* The caller has found an ANY SubLink at the top level of one of the query's
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index c1faf41..5ffdd8f 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -518,7 +518,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
Bit ConstBit BitWithLength BitWithoutLength
%type <str> character
%type <str> extract_arg
-%type <boolean> opt_varying opt_timezone opt_no_inherit
+%type <boolean> opt_varying opt_timezone opt_no_inherit opt_materialized
%type <ival> Iconst SignedIconst
%type <str> Sconst comment_text notify_payload
@@ -11322,17 +11322,23 @@ cte_list:
| cte_list ',' common_table_expr { $$ = lappend($1, $3); }
;
-common_table_expr: name opt_name_list AS '(' PreparableStmt ')'
+common_table_expr: name opt_name_list AS opt_materialized '(' PreparableStmt ')'
{
CommonTableExpr *n = makeNode(CommonTableExpr);
n->ctename = $1;
n->aliascolnames = $2;
- n->ctequery = $5;
+ n->ctematerialized = $4;
+ n->ctequery = $6;
n->location = @1;
$$ = (Node *) n;
}
;
+opt_materialized:
+ MATERIALIZED { $$ = true; }
+ | /*EMPTY*/ { $$ = false; }
+ ;
+
opt_with_clause:
with_clause { $$ = $1; }
| /*EMPTY*/ { $$ = NULL; }
@@ -16214,6 +16220,7 @@ makeRecursiveViewSelect(char *relname, List *aliases, Node *query)
/* create common table expression */
cte->ctename = relname;
cte->aliascolnames = aliases;
+ cte->ctematerialized = false;
cte->ctequery = query;
cte->location = -1;
diff --git a/src/backend/utils/adt/ruleutils.c b/src/backend/utils/adt/ruleutils.c
index 302df16..eb0010d 100644
--- a/src/backend/utils/adt/ruleutils.c
+++ b/src/backend/utils/adt/ruleutils.c
@@ -5166,7 +5166,10 @@ get_with_clause(Query *query, deparse_context *context)
}
appendStringInfoChar(buf, ')');
}
- appendStringInfoString(buf, " AS (");
+ appendStringInfoString(buf, " AS ");
+ if (cte->ctematerialized)
+ appendStringInfoString(buf, "MATERIALIZED ");
+ appendStringInfoChar(buf, '(');
if (PRETTY_INDENT(context))
appendContextKeyword(context, "", 0, 0, 0);
get_query_def((Query *) cte->ctequery, buf, context->namespaces, NULL,
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index addc2c2..136f72c 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -1404,6 +1404,7 @@ typedef struct CommonTableExpr
NodeTag type;
char *ctename; /* query name (never qualified) */
List *aliascolnames; /* optional list of column names */
+ bool ctematerialized; /* is this an optimization fence? */
/* SelectStmt/InsertStmt/etc before parse analysis, Query afterwards: */
Node *ctequery; /* the CTE's subquery */
int location; /* token location, or -1 if unknown */
diff --git a/src/test/regress/expected/rowsecurity.out b/src/test/regress/expected/rowsecurity.out
index 1d12b01..2e17049 100644
--- a/src/test/regress/expected/rowsecurity.out
+++ b/src/test/regress/expected/rowsecurity.out
@@ -2179,7 +2179,7 @@ EXPLAIN (COSTS OFF) EXECUTE plancache_test;
Filter: (((a % 2) = 0) AND f_leak(b))
(2 rows)
-PREPARE plancache_test2 AS WITH q AS (SELECT * FROM z1 WHERE f_leak(b)) SELECT * FROM q,z2;
+PREPARE plancache_test2 AS WITH q AS MATERIALIZED (SELECT * FROM z1 WHERE f_leak(b)) SELECT * FROM q,z2;
EXPLAIN (COSTS OFF) EXECUTE plancache_test2;
QUERY PLAN
-------------------------------------------------
@@ -2192,7 +2192,7 @@ EXPLAIN (COSTS OFF) EXECUTE plancache_test2;
-> Seq Scan on z2
(7 rows)
-PREPARE plancache_test3 AS WITH q AS (SELECT * FROM z2) SELECT * FROM q,z1 WHERE f_leak(z1.b);
+PREPARE plancache_test3 AS WITH q AS MATERIALIZED (SELECT * FROM z2) SELECT * FROM q,z1 WHERE f_leak(z1.b);
EXPLAIN (COSTS OFF) EXECUTE plancache_test3;
QUERY PLAN
-----------------------------------------------------
@@ -2826,7 +2826,7 @@ ALTER TABLE t1 ENABLE ROW LEVEL SECURITY;
GRANT ALL ON t1 TO regress_rls_bob;
INSERT INTO t1 (SELECT x, md5(x::text) FROM generate_series(0,20) x);
SET SESSION AUTHORIZATION regress_rls_bob;
-WITH cte1 AS (SELECT * FROM t1 WHERE f_leak(b)) SELECT * FROM cte1;
+WITH cte1 AS MATERIALIZED (SELECT * FROM t1 WHERE f_leak(b)) SELECT * FROM cte1;
NOTICE: f_leak => cfcd208495d565ef66e7dff9f98764da
NOTICE: f_leak => c81e728d9d4c2f636f067f89cc14862c
NOTICE: f_leak => a87ff679a2f3e71d9181a67b7542122c
@@ -2853,7 +2853,8 @@ NOTICE: f_leak => 98f13708210194c475687be6106a3b84
20 | 98f13708210194c475687be6106a3b84
(11 rows)
-EXPLAIN (COSTS OFF) WITH cte1 AS (SELECT * FROM t1 WHERE f_leak(b)) SELECT * FROM cte1;
+EXPLAIN (COSTS OFF)
+WITH cte1 AS MATERIALIZED (SELECT * FROM t1 WHERE f_leak(b)) SELECT * FROM cte1;
QUERY PLAN
-------------------------------------------------
CTE Scan on cte1
diff --git a/src/test/regress/expected/rowtypes.out b/src/test/regress/expected/rowtypes.out
index d6a1a33..14757b4 100644
--- a/src/test/regress/expected/rowtypes.out
+++ b/src/test/regress/expected/rowtypes.out
@@ -1037,7 +1037,7 @@ from (values (1,row(1,2)), (1,row(null,null)), (1,null),
(6 rows)
explain (verbose, costs off)
-with r(a,b) as
+with r(a,b) as materialized
(values (1,row(1,2)), (1,row(null,null)), (1,null),
(null,row(1,2)), (null,row(null,null)), (null,null) )
select r, r is null as isnull, r is not null as isnotnull from r;
@@ -1050,7 +1050,7 @@ select r, r is null as isnull, r is not null as isnotnull from r;
Output: "*VALUES*".column1, "*VALUES*".column2
(5 rows)
-with r(a,b) as
+with r(a,b) as materialized
(values (1,row(1,2)), (1,row(null,null)), (1,null),
(null,row(1,2)), (null,row(null,null)), (null,null) )
select r, r is null as isnull, r is not null as isnotnull from r;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index e384cd2..bb10e27 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -3108,7 +3108,7 @@ explain (costs off) INSERT INTO hats VALUES ('h8', 'forbidden') RETURNING *;
(5 rows)
-- ensure upserting into a rule, with a CTE (different offsets!) works
-WITH data(hat_name, hat_color) AS (
+WITH data(hat_name, hat_color) AS MATERIALIZED (
VALUES ('h8', 'green'),
('h9', 'blue'),
('h7', 'forbidden')
@@ -3122,7 +3122,8 @@ RETURNING *;
h9 | blue
(2 rows)
-EXPLAIN (costs off) WITH data(hat_name, hat_color) AS (
+EXPLAIN (costs off)
+WITH data(hat_name, hat_color) AS MATERIALIZED (
VALUES ('h8', 'green'),
('h9', 'blue'),
('h7', 'forbidden')
diff --git a/src/test/regress/expected/subselect.out b/src/test/regress/expected/subselect.out
index 588d069..e2afd6f 100644
--- a/src/test/regress/expected/subselect.out
+++ b/src/test/regress/expected/subselect.out
@@ -1154,3 +1154,117 @@ fetch backward all in c1;
(2 rows)
commit;
+--
+-- Tests for CTE inlining behavior
+--
+-- Basic subquery that can be inlined
+explain (verbose, costs off)
+with x as (select * from (select f1 from subselect_tbl) ss)
+select * from x where f1 = 1;
+ QUERY PLAN
+----------------------------------
+ Seq Scan on public.subselect_tbl
+ Output: subselect_tbl.f1
+ Filter: (subselect_tbl.f1 = 1)
+(3 rows)
+
+-- Explicitly request materialization
+explain (verbose, costs off)
+with x as materialized (select * from (select f1 from subselect_tbl) ss)
+select * from x where f1 = 1;
+ QUERY PLAN
+------------------------------------------
+ CTE Scan on x
+ Output: x.f1
+ Filter: (x.f1 = 1)
+ CTE x
+ -> Seq Scan on public.subselect_tbl
+ Output: subselect_tbl.f1
+(6 rows)
+
+-- Stable functions are safe to inline
+explain (verbose, costs off)
+with x as (select * from (select f1, now() from subselect_tbl) ss)
+select * from x where f1 = 1;
+ QUERY PLAN
+-----------------------------------
+ Seq Scan on public.subselect_tbl
+ Output: subselect_tbl.f1, now()
+ Filter: (subselect_tbl.f1 = 1)
+(3 rows)
+
+-- Volatile functions prevent inlining
+explain (verbose, costs off)
+with x as (select * from (select f1, random() from subselect_tbl) ss)
+select * from x where f1 = 1;
+ QUERY PLAN
+----------------------------------------------
+ CTE Scan on x
+ Output: x.f1, x.random
+ Filter: (x.f1 = 1)
+ CTE x
+ -> Seq Scan on public.subselect_tbl
+ Output: subselect_tbl.f1, random()
+(6 rows)
+
+-- SELECT FOR UPDATE cannot be inlined
+explain (verbose, costs off)
+with x as (select * from (select f1 from subselect_tbl for update) ss)
+select * from x where f1 = 1;
+ QUERY PLAN
+--------------------------------------------------------------------
+ CTE Scan on x
+ Output: x.f1
+ Filter: (x.f1 = 1)
+ CTE x
+ -> Subquery Scan on ss
+ Output: ss.f1
+ -> LockRows
+ Output: subselect_tbl.f1, subselect_tbl.ctid
+ -> Seq Scan on public.subselect_tbl
+ Output: subselect_tbl.f1, subselect_tbl.ctid
+(10 rows)
+
+-- Check handling of outer references
+explain (verbose, costs off)
+with x as (select * from int4_tbl)
+select * from (with y as (select * from x) select * from y) ss;
+ QUERY PLAN
+-----------------------------
+ Seq Scan on public.int4_tbl
+ Output: int4_tbl.f1
+(2 rows)
+
+explain (verbose, costs off)
+with x as materialized (select * from int4_tbl)
+select * from (with y as (select * from x) select * from y) ss;
+ QUERY PLAN
+-------------------------------------
+ CTE Scan on x
+ Output: x.f1
+ CTE x
+ -> Seq Scan on public.int4_tbl
+ Output: int4_tbl.f1
+(5 rows)
+
+-- Ensure that we inline the currect CTE when there are
+-- multiple CTEs with the same name
+explain (verbose, costs off)
+with x as (select 1 as y)
+select * from (with x as (select 2 as y) select * from x) ss;
+ QUERY PLAN
+-------------
+ Result
+ Output: 2
+(2 rows)
+
+-- Row marks are not pushed into CTEs
+explain (verbose, costs off)
+with x as (select * from subselect_tbl)
+select * from x for update;
+ QUERY PLAN
+----------------------------------------------------------------
+ Seq Scan on public.subselect_tbl
+ Output: subselect_tbl.f1, subselect_tbl.f2, subselect_tbl.f3
+(2 rows)
+
diff --git a/src/test/regress/sql/rowsecurity.sql b/src/test/regress/sql/rowsecurity.sql
index 38e9b38..52da276 100644
--- a/src/test/regress/sql/rowsecurity.sql
+++ b/src/test/regress/sql/rowsecurity.sql
@@ -840,10 +840,10 @@ EXPLAIN (COSTS OFF) SELECT * FROM z1 WHERE f_leak(b);
PREPARE plancache_test AS SELECT * FROM z1 WHERE f_leak(b);
EXPLAIN (COSTS OFF) EXECUTE plancache_test;
-PREPARE plancache_test2 AS WITH q AS (SELECT * FROM z1 WHERE f_leak(b)) SELECT * FROM q,z2;
+PREPARE plancache_test2 AS WITH q AS MATERIALIZED (SELECT * FROM z1 WHERE f_leak(b)) SELECT * FROM q,z2;
EXPLAIN (COSTS OFF) EXECUTE plancache_test2;
-PREPARE plancache_test3 AS WITH q AS (SELECT * FROM z2) SELECT * FROM q,z1 WHERE f_leak(z1.b);
+PREPARE plancache_test3 AS WITH q AS MATERIALIZED (SELECT * FROM z2) SELECT * FROM q,z1 WHERE f_leak(z1.b);
EXPLAIN (COSTS OFF) EXECUTE plancache_test3;
SET ROLE regress_rls_group1;
@@ -1071,8 +1071,9 @@ INSERT INTO t1 (SELECT x, md5(x::text) FROM generate_series(0,20) x);
SET SESSION AUTHORIZATION regress_rls_bob;
-WITH cte1 AS (SELECT * FROM t1 WHERE f_leak(b)) SELECT * FROM cte1;
-EXPLAIN (COSTS OFF) WITH cte1 AS (SELECT * FROM t1 WHERE f_leak(b)) SELECT * FROM cte1;
+WITH cte1 AS MATERIALIZED (SELECT * FROM t1 WHERE f_leak(b)) SELECT * FROM cte1;
+EXPLAIN (COSTS OFF)
+WITH cte1 AS MATERIALIZED (SELECT * FROM t1 WHERE f_leak(b)) SELECT * FROM cte1;
WITH cte1 AS (UPDATE t1 SET a = a + 1 RETURNING *) SELECT * FROM cte1; --fail
WITH cte1 AS (UPDATE t1 SET a = a RETURNING *) SELECT * FROM cte1; --ok
diff --git a/src/test/regress/sql/rowtypes.sql b/src/test/regress/sql/rowtypes.sql
index e6d3898..b3b0774 100644
--- a/src/test/regress/sql/rowtypes.sql
+++ b/src/test/regress/sql/rowtypes.sql
@@ -428,12 +428,12 @@ from (values (1,row(1,2)), (1,row(null,null)), (1,null),
(null,row(1,2)), (null,row(null,null)), (null,null) ) r(a,b);
explain (verbose, costs off)
-with r(a,b) as
+with r(a,b) as materialized
(values (1,row(1,2)), (1,row(null,null)), (1,null),
(null,row(1,2)), (null,row(null,null)), (null,null) )
select r, r is null as isnull, r is not null as isnotnull from r;
-with r(a,b) as
+with r(a,b) as materialized
(values (1,row(1,2)), (1,row(null,null)), (1,null),
(null,row(1,2)), (null,row(null,null)), (null,null) )
select r, r is null as isnull, r is not null as isnotnull from r;
diff --git a/src/test/regress/sql/rules.sql b/src/test/regress/sql/rules.sql
index f4ee30e..6666951 100644
--- a/src/test/regress/sql/rules.sql
+++ b/src/test/regress/sql/rules.sql
@@ -1132,7 +1132,7 @@ SELECT tablename, rulename, definition FROM pg_rules
explain (costs off) INSERT INTO hats VALUES ('h8', 'forbidden') RETURNING *;
-- ensure upserting into a rule, with a CTE (different offsets!) works
-WITH data(hat_name, hat_color) AS (
+WITH data(hat_name, hat_color) AS MATERIALIZED (
VALUES ('h8', 'green'),
('h9', 'blue'),
('h7', 'forbidden')
@@ -1140,7 +1140,8 @@ WITH data(hat_name, hat_color) AS (
INSERT INTO hats
SELECT * FROM data
RETURNING *;
-EXPLAIN (costs off) WITH data(hat_name, hat_color) AS (
+EXPLAIN (costs off)
+WITH data(hat_name, hat_color) AS MATERIALIZED (
VALUES ('h8', 'green'),
('h9', 'blue'),
('h7', 'forbidden')
diff --git a/src/test/regress/sql/subselect.sql b/src/test/regress/sql/subselect.sql
index 843f511..3dba35e 100644
--- a/src/test/regress/sql/subselect.sql
+++ b/src/test/regress/sql/subselect.sql
@@ -625,3 +625,52 @@ move forward all in c1;
fetch backward all in c1;
commit;
+
+--
+-- Tests for CTE inlining behavior
+--
+
+-- Basic subquery that can be inlined
+explain (verbose, costs off)
+with x as (select * from (select f1 from subselect_tbl) ss)
+select * from x where f1 = 1;
+
+-- Explicitly request materialization
+explain (verbose, costs off)
+with x as materialized (select * from (select f1 from subselect_tbl) ss)
+select * from x where f1 = 1;
+
+-- Stable functions are safe to inline
+explain (verbose, costs off)
+with x as (select * from (select f1, now() from subselect_tbl) ss)
+select * from x where f1 = 1;
+
+-- Volatile functions prevent inlining
+explain (verbose, costs off)
+with x as (select * from (select f1, random() from subselect_tbl) ss)
+select * from x where f1 = 1;
+
+-- SELECT FOR UPDATE cannot be inlined
+explain (verbose, costs off)
+with x as (select * from (select f1 from subselect_tbl for update) ss)
+select * from x where f1 = 1;
+
+-- Check handling of outer references
+explain (verbose, costs off)
+with x as (select * from int4_tbl)
+select * from (with y as (select * from x) select * from y) ss;
+
+explain (verbose, costs off)
+with x as materialized (select * from int4_tbl)
+select * from (with y as (select * from x) select * from y) ss;
+
+-- Ensure that we inline the currect CTE when there are
+-- multiple CTEs with the same name
+explain (verbose, costs off)
+with x as (select 1 as y)
+select * from (with x as (select 2 as y) select * from x) ss;
+
+-- Row marks are not pushed into CTEs
+explain (verbose, costs off)
+with x as (select * from subselect_tbl)
+select * from x for update;
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-22 03:47 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-25 22:22 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
@ 2019-01-26 22:55 ` Tom Lane <tgl@sss.pgh.pa.us>
2019-01-27 13:22 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-27 18:38 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
1 sibling, 2 replies; 112+ messages in thread
From: Tom Lane @ 2019-01-26 22:55 UTC (permalink / raw)
To: Andreas Karlsson <andreas@proxel.se>; +Cc: Andrew Gierth <andrew@tao11.riddles.org.uk>; David Fetter <david@fetter.org>; Thomas Munro <thomas.munro@enterprisedb.com>; pgsql-hackers
I wrote:
> Therefore, I'm reversing my previous opinion that we should not have
> an explicit NOT MATERIALIZED option. I think we should add that, and
> the behavior ought to be:
> * No option given: inline if there's exactly one reference.
> * With MATERIALIZED: never inline.
> * With NOT MATERIALIZED: inline regardless of the number of references.
> (Obviously, we should not inline if there's RECURSIVE or the CTE
> potentially has side-effects, regardless of the user option;
> I don't think those cases are up for debate.)
Hearing no immediate pushback on that proposal, I went ahead and made
a version of the patch that does it like that, as attached. I also took
a stab at documenting it fully.
I was interested to find, while writing the docs, that it's a real
struggle to invent plausible reasons to write MATERIALIZED given the
above specification. You pretty much have to have lied to the planner,
eg by making a volatile function that's not marked volatile, before
there's a real need for that. Am I missing something? If I'm not,
then we're in a really good place backwards-compatibility-wise,
because the new default behavior shouldn't break any cases where people
weren't cheating.
regards, tom lane
Attachments:
[text/x-diff] inlining-ctes-v10.patch (40.9K, ../../31065.1548543317@sss.pgh.pa.us/2-inlining-ctes-v10.patch)
download | inline diff:
diff --git a/contrib/pg_stat_statements/pg_stat_statements.c b/contrib/pg_stat_statements/pg_stat_statements.c
index f177eba..6d456f6 100644
--- a/contrib/pg_stat_statements/pg_stat_statements.c
+++ b/contrib/pg_stat_statements/pg_stat_statements.c
@@ -2925,6 +2925,7 @@ JumbleExpr(pgssJumbleState *jstate, Node *node)
/* we store the string name because RTE_CTE RTEs need it */
APP_JUMB_STRING(cte->ctename);
+ APP_JUMB(cte->ctematerialized);
JumbleQuery(jstate, castNode(Query, cte->ctequery));
}
break;
diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out
index bb92d9d..8c26dd1 100644
--- a/contrib/postgres_fdw/expected/postgres_fdw.out
+++ b/contrib/postgres_fdw/expected/postgres_fdw.out
@@ -1888,7 +1888,7 @@ SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) ORDER BY t1.c3, t
-- join in CTE
EXPLAIN (VERBOSE, COSTS OFF)
-WITH t (c1_1, c1_3, c2_1) AS (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
+WITH t (c1_1, c1_3, c2_1) AS MATERIALIZED (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
QUERY PLAN
-------------------------------------------------------------------------------------------------------------------------------------
Limit
@@ -1905,7 +1905,7 @@ WITH t (c1_1, c1_3, c2_1) AS (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2
Output: t.c1_1, t.c2_1, t.c1_3
(12 rows)
-WITH t (c1_1, c1_3, c2_1) AS (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
+WITH t (c1_1, c1_3, c2_1) AS MATERIALIZED (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
c1_1 | c2_1
------+------
101 | 101
diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql
index f438165..56602a1 100644
--- a/contrib/postgres_fdw/sql/postgres_fdw.sql
+++ b/contrib/postgres_fdw/sql/postgres_fdw.sql
@@ -493,8 +493,8 @@ SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) ORDER BY t1.c3, t
SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) ORDER BY t1.c3, t1.c1 OFFSET 100 LIMIT 10 FOR SHARE;
-- join in CTE
EXPLAIN (VERBOSE, COSTS OFF)
-WITH t (c1_1, c1_3, c2_1) AS (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
-WITH t (c1_1, c1_3, c2_1) AS (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
+WITH t (c1_1, c1_3, c2_1) AS MATERIALIZED (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
+WITH t (c1_1, c1_3, c2_1) AS MATERIALIZED (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) SELECT c1_1, c2_1 FROM t ORDER BY c1_3, c1_1 OFFSET 100 LIMIT 10;
-- ctid with whole-row reference
EXPLAIN (VERBOSE, COSTS OFF)
SELECT t1.ctid, t1, t2, t1.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) ORDER BY t1.c3, t1.c1 OFFSET 100 LIMIT 10;
diff --git a/doc/src/sgml/queries.sgml b/doc/src/sgml/queries.sgml
index 88bc189..2225255 100644
--- a/doc/src/sgml/queries.sgml
+++ b/doc/src/sgml/queries.sgml
@@ -2199,16 +2199,19 @@ SELECT n FROM t LIMIT 100;
</para>
<para>
- A useful property of <literal>WITH</literal> queries is that they are evaluated
- only once per execution of the parent query, even if they are referred to
- more than once by the parent query or sibling <literal>WITH</literal> queries.
+ A useful property of <literal>WITH</literal> queries is that they are
+ normally evaluated only once per execution of the parent query, even if
+ they are referred to more than once by the parent query or
+ sibling <literal>WITH</literal> queries.
Thus, expensive calculations that are needed in multiple places can be
placed within a <literal>WITH</literal> query to avoid redundant work. Another
possible application is to prevent unwanted multiple evaluations of
functions with side-effects.
- However, the other side of this coin is that the optimizer is less able to
- push restrictions from the parent query down into a <literal>WITH</literal> query
- than an ordinary subquery. The <literal>WITH</literal> query will generally be
+ However, the other side of this coin is that the optimizer is not able to
+ push restrictions from the parent query down into a multiply-referenced
+ <literal>WITH</literal> query, since that might affect all uses of the
+ <literal>WITH</literal> query's output when it should affect only one.
+ The multiply-referenced <literal>WITH</literal> query will be
evaluated as written, without suppression of rows that the parent query
might discard afterwards. (But, as mentioned above, evaluation might stop
early if the reference(s) to the query demand only a limited number of
@@ -2216,6 +2219,75 @@ SELECT n FROM t LIMIT 100;
</para>
<para>
+ However, if a <literal>WITH</literal> query is non-recursive and
+ side-effect-free (that is, it is a <literal>SELECT</literal> containing
+ no volatile functions) then it can be folded into the parent query,
+ allowing joint optimization of the two query levels. By default, this
+ happens if the parent query references the <literal>WITH</literal> query
+ just once, but not if it references the <literal>WITH</literal> query
+ more than once. You can override that decision by
+ specifying <literal>MATERIALIZED</literal> to force separate calculation
+ of the <literal>WITH</literal> query, or by specifying <literal>NOT
+ MATERIALIZED</literal> to force it to be merged into the parent query.
+ The latter choice risks duplicate computation of
+ the <literal>WITH</literal> query, but it can still give a net savings if
+ each usage of the <literal>WITH</literal> query needs only a small part
+ of the <literal>WITH</literal> query's full output.
+ </para>
+
+ <para>
+ A simple example of these rules is
+<programlisting>
+WITH w AS (
+ SELECT * FROM big_table
+)
+SELECT * FROM w WHERE key = 123;
+</programlisting>
+ This <literal>WITH</literal> query will be folded, producing the same
+ execution plan as
+<programlisting>
+SELECT * FROM big_table WHERE key = 123;
+</programlisting>
+ In particular, if there's an index on <structfield>key</structfield>,
+ it will probably be used to fetch just the rows having <literal>key =
+ 123</literal>. On the other hand, in
+<programlisting>
+WITH w AS (
+ SELECT * FROM big_table
+)
+SELECT * FROM w AS w1 JOIN w AS w2 ON w1.key = w2.ref
+WHERE w2.key = 123;
+</programlisting>
+ the <literal>WITH</literal> query will be materialized, producing a
+ temporary copy of <structname>big_table</structname> that is then
+ joined with itself — without benefit of any index. This query
+ will be executed much more efficiently if written as
+<programlisting>
+WITH w AS NOT MATERIALIZED (
+ SELECT * FROM big_table
+)
+SELECT * FROM w AS w1 JOIN w AS w2 ON w1.key = w2.ref
+WHERE w2.key = 123;
+</programlisting>
+ so that the parent query's restrictions can be applied directly
+ to scans of <structname>big_table</structname>.
+ </para>
+
+ <para>
+ An example where <literal>NOT MATERIALIZED</literal> could be
+ undesirable is
+<programlisting>
+WITH w AS (
+ SELECT key, very_expensive_function(val) as f FROM some_table
+)
+SELECT * FROM w AS w1 JOIN w AS w2 ON w1.f = w2.f;
+</programlisting>
+ Here, materialization of the <literal>WITH</literal> query ensures
+ that <function>very_expensive_function</function> is evaluated only
+ once per table row, not twice.
+ </para>
+
+ <para>
The examples above only show <literal>WITH</literal> being used with
<command>SELECT</command>, but it can be attached in the same way to
<command>INSERT</command>, <command>UPDATE</command>, or <command>DELETE</command>.
diff --git a/doc/src/sgml/ref/select.sgml b/doc/src/sgml/ref/select.sgml
index 4db8142..7035c95 100644
--- a/doc/src/sgml/ref/select.sgml
+++ b/doc/src/sgml/ref/select.sgml
@@ -72,7 +72,7 @@ SELECT [ ALL | DISTINCT [ ON ( <replaceable class="parameter">expression</replac
<phrase>and <replaceable class="parameter">with_query</replaceable> is:</phrase>
- <replaceable class="parameter">with_query_name</replaceable> [ ( <replaceable class="parameter">column_name</replaceable> [, ...] ) ] AS ( <replaceable class="parameter">select</replaceable> | <replaceable class="parameter">values</replaceable> | <replaceable class="parameter">insert</replaceable> | <replaceable class="parameter">update</replaceable> | <replaceable class="parameter">delete</replaceable> )
+ <replaceable class="parameter">with_query_name</replaceable> [ ( <replaceable class="parameter">column_name</replaceable> [, ...] ) ] AS [ [ NOT ] MATERIALIZED ] ( <replaceable class="parameter">select</replaceable> | <replaceable class="parameter">values</replaceable> | <replaceable class="parameter">insert</replaceable> | <replaceable class="parameter">update</replaceable> | <replaceable class="parameter">delete</replaceable> )
TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ]
</synopsis>
@@ -93,7 +93,8 @@ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ]
These effectively serve as temporary tables that can be referenced
in the <literal>FROM</literal> list. A <literal>WITH</literal> query
that is referenced more than once in <literal>FROM</literal> is
- computed only once.
+ computed only once,
+ unless specified otherwise with <literal>NOT MATERIALIZED</literal>.
(See <xref linkend="sql-with" endterm="sql-with-title"/> below.)
</para>
</listitem>
@@ -273,8 +274,17 @@ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ]
</para>
<para>
+ The primary query and the <literal>WITH</literal> queries are all
+ (notionally) executed at the same time. This implies that the effects of
+ a data-modifying statement in <literal>WITH</literal> cannot be seen from
+ other parts of the query, other than by reading its <literal>RETURNING</literal>
+ output. If two such data-modifying statements attempt to modify the same
+ row, the results are unspecified.
+ </para>
+
+ <para>
A key property of <literal>WITH</literal> queries is that they
- are evaluated only once per execution of the primary query,
+ are normally evaluated only once per execution of the primary query,
even if the primary query refers to them more than once.
In particular, data-modifying statements are guaranteed to be
executed once and only once, regardless of whether the primary query
@@ -282,12 +292,35 @@ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ]
</para>
<para>
- The primary query and the <literal>WITH</literal> queries are all
- (notionally) executed at the same time. This implies that the effects of
- a data-modifying statement in <literal>WITH</literal> cannot be seen from
- other parts of the query, other than by reading its <literal>RETURNING</literal>
- output. If two such data-modifying statements attempt to modify the same
- row, the results are unspecified.
+ However, a <literal>WITH</literal> query can be marked
+ <literal>NOT MATERIALIZED</literal> to remove this guarantee. In that
+ case, the <literal>WITH</literal> query can be folded into the primary
+ query much as though it were a simple sub-<literal>SELECT</literal> in
+ the primary query's <literal>FROM</literal> clause. This results in
+ duplicate computations if the primary query refers to
+ that <literal>WITH</literal> query more than once; but if each such use
+ requires only a few rows of the <literal>WITH</literal> query's total
+ output, <literal>NOT MATERIALIZED</literal> can provide a net savings by
+ allowing the queries to be optimized jointly.
+ <literal>NOT MATERIALIZED</literal> is ignored if it is attached to
+ a <literal>WITH</literal> query that is recursive or is not
+ side-effect-free (i.e., is not a plain <literal>SELECT</literal>
+ containing no volatile functions).
+ </para>
+
+ <para>
+ By default, a side-effect-free <literal>WITH</literal> query is folded
+ into the primary query if it is used exactly once in the primary
+ query's <literal>FROM</literal> clause. This allows joint optimization
+ of the two query levels in situations where that should be semantically
+ invisible. However, such folding can be prevented by marking the
+ <literal>WITH</literal> query as <literal>MATERIALIZED</literal>.
+ That might be useful, for example, if the <literal>WITH</literal> query
+ has side-effects that are hidden from the planner.
+ <productname>PostgreSQL</productname> versions before v12 never did
+ such folding, so another possible reason for
+ using <literal>MATERIALIZED</literal> is to duplicate the pre-v12
+ behavior.
</para>
<para>
@@ -2087,6 +2120,12 @@ SELECT distributors.* WHERE distributors.name = 'Westward';
<para>
<literal>ROWS FROM( ... )</literal> is an extension of the SQL standard.
</para>
+
+ <para>
+ The <literal>MATERIALIZED</literal> and <literal>NOT
+ MATERIALIZED</literal> options of <literal>WITH</literal> are extensions
+ of the SQL standard.
+ </para>
</refsect2>
</refsect1>
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 3eb7e95..3b6e206 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2536,6 +2536,7 @@ _copyCommonTableExpr(const CommonTableExpr *from)
COPY_STRING_FIELD(ctename);
COPY_NODE_FIELD(aliascolnames);
+ COPY_SCALAR_FIELD(ctematerialized);
COPY_NODE_FIELD(ctequery);
COPY_LOCATION_FIELD(location);
COPY_SCALAR_FIELD(cterecursive);
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index 5c4fa7d..e7b0fae 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2791,6 +2791,7 @@ _equalCommonTableExpr(const CommonTableExpr *a, const CommonTableExpr *b)
{
COMPARE_STRING_FIELD(ctename);
COMPARE_NODE_FIELD(aliascolnames);
+ COMPARE_SCALAR_FIELD(ctematerialized);
COMPARE_NODE_FIELD(ctequery);
COMPARE_LOCATION_FIELD(location);
COMPARE_SCALAR_FIELD(cterecursive);
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 0fde876..9035575 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2980,6 +2980,7 @@ _outCommonTableExpr(StringInfo str, const CommonTableExpr *node)
WRITE_STRING_FIELD(ctename);
WRITE_NODE_FIELD(aliascolnames);
+ WRITE_ENUM_FIELD(ctematerialized, CTEMaterialize);
WRITE_NODE_FIELD(ctequery);
WRITE_LOCATION_FIELD(location);
WRITE_BOOL_FIELD(cterecursive);
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index ec6f256..be30894 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -418,6 +418,7 @@ _readCommonTableExpr(void)
READ_STRING_FIELD(ctename);
READ_NODE_FIELD(aliascolnames);
+ READ_ENUM_FIELD(ctematerialized, CTEMaterialize);
READ_NODE_FIELD(ctequery);
READ_LOCATION_FIELD(location);
READ_BOOL_FIELD(cterecursive);
diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c
index 4465f00..67f276a 100644
--- a/src/backend/optimizer/plan/planner.c
+++ b/src/backend/optimizer/plan/planner.c
@@ -645,8 +645,8 @@ subquery_planner(PlannerGlobal *glob, Query *parse,
root->partColsUpdated = false;
/*
- * If there is a WITH list, process each WITH query and build an initplan
- * SubPlan structure for it.
+ * If there is a WITH list, process each WITH query and either convert it
+ * to RTE_SUBQUERY RTE(s) or build an initplan SubPlan structure for it.
*/
if (parse->cteList)
SS_process_ctes(root);
diff --git a/src/backend/optimizer/plan/subselect.c b/src/backend/optimizer/plan/subselect.c
index 64272dd..dde3715 100644
--- a/src/backend/optimizer/plan/subselect.c
+++ b/src/backend/optimizer/plan/subselect.c
@@ -57,6 +57,14 @@ typedef struct finalize_primnode_context
Bitmapset *paramids; /* Non-local PARAM_EXEC paramids found */
} finalize_primnode_context;
+typedef struct inline_cte_walker_context
+{
+ const char *ctename; /* name and relative level of target CTE */
+ int levelsup;
+ int refcount; /* number of remaining references */
+ Query *ctequery; /* query to substitute */
+} inline_cte_walker_context;
+
static Node *build_subplan(PlannerInfo *root, Plan *plan, PlannerInfo *subroot,
List *plan_params,
@@ -75,6 +83,10 @@ static Node *convert_testexpr_mutator(Node *node,
static bool subplan_is_hashable(Plan *plan);
static bool testexpr_is_hashable(Node *testexpr);
static bool hash_ok_operator(OpExpr *expr);
+static bool contain_dml(Node *node);
+static bool contain_dml_walker(Node *node, void *context);
+static void inline_cte(PlannerInfo *root, CommonTableExpr *cte);
+static bool inline_cte_walker(Node *node, inline_cte_walker_context *context);
static bool simplify_EXISTS_query(PlannerInfo *root, Query *query);
static Query *convert_EXISTS_to_ANY(PlannerInfo *root, Query *subselect,
Node **testexpr, List **paramIds);
@@ -804,10 +816,13 @@ hash_ok_operator(OpExpr *expr)
/*
* SS_process_ctes: process a query's WITH list
*
- * We plan each interesting WITH item and convert it to an initplan.
+ * Consider each CTE in the WITH list and either ignore it (if it's an
+ * unreferenced SELECT), "inline" it to create a regular sub-SELECT-in-FROM,
+ * or convert it to an initplan.
+ *
* A side effect is to fill in root->cte_plan_ids with a list that
* parallels root->parse->cteList and provides the subplan ID for
- * each CTE's initplan.
+ * each CTE's initplan, or a dummy ID (-1) if we didn't make an initplan.
*/
void
SS_process_ctes(PlannerInfo *root)
@@ -839,6 +854,44 @@ SS_process_ctes(PlannerInfo *root)
}
/*
+ * Consider inlining the CTE (creating RTE_SUBQUERY RTE(s)) instead of
+ * implementing it as a separately-planned CTE.
+ *
+ * We cannot inline if any of these conditions hold:
+ *
+ * 1. The user said not to (the CTEMaterializeAlways option).
+ *
+ * 2. The CTE is recursive.
+ *
+ * 3. The CTE has side-effects; this includes either not being a plain
+ * SELECT, or containing volatile functions. Inlining might change
+ * the side-effects, which would be bad.
+ *
+ * Otherwise, we have an option whether to inline or not. That should
+ * always be a win if there's just a single reference, but if the CTE
+ * is multiply-referenced then it's unclear: inlining adds duplicate
+ * computations, but the ability to absorb restrictions from the outer
+ * query level could outweigh that. We do not have nearly enough
+ * information at this point to tell whether that's true, so we let
+ * the user express a preference. Our default behavior is to inline
+ * only singly-referenced CTEs, but a CTE marked CTEMaterializeNever
+ * will be inlined even if multiply referenced.
+ */
+ if ((cte->ctematerialized == CTEMaterializeNever ||
+ (cte->ctematerialized == CTEMaterializeDefault &&
+ cte->cterefcount == 1)) &&
+ !cte->cterecursive &&
+ cmdType == CMD_SELECT &&
+ !contain_dml(cte->ctequery) &&
+ !contain_volatile_functions(cte->ctequery))
+ {
+ inline_cte(root, cte);
+ /* Make a dummy entry in cte_plan_ids */
+ root->cte_plan_ids = lappend_int(root->cte_plan_ids, -1);
+ continue;
+ }
+
+ /*
* Copy the source Query node. Probably not necessary, but let's keep
* this similar to make_subplan.
*/
@@ -935,6 +988,127 @@ SS_process_ctes(PlannerInfo *root)
}
/*
+ * contain_dml: is any subquery not a plain SELECT?
+ *
+ * We reject SELECT FOR UPDATE/SHARE as well as INSERT etc.
+ */
+static bool
+contain_dml(Node *node)
+{
+ return contain_dml_walker(node, NULL);
+}
+
+static bool
+contain_dml_walker(Node *node, void *context)
+{
+ if (node == NULL)
+ return false;
+ if (IsA(node, Query))
+ {
+ Query *query = (Query *) node;
+
+ if (query->commandType != CMD_SELECT ||
+ query->rowMarks != NIL)
+ return true;
+
+ return query_tree_walker(query, contain_dml_walker, context, 0);
+ }
+ return expression_tree_walker(node, contain_dml_walker, context);
+}
+
+/*
+ * inline_cte: convert RTE_CTE references to given CTE into RTE_SUBQUERYs
+ */
+static void
+inline_cte(PlannerInfo *root, CommonTableExpr *cte)
+{
+ struct inline_cte_walker_context context;
+
+ context.ctename = cte->ctename;
+ /* Start at levelsup = -1 because we'll immediately increment it */
+ context.levelsup = -1;
+ context.refcount = cte->cterefcount;
+ context.ctequery = castNode(Query, cte->ctequery);
+
+ (void) inline_cte_walker((Node *) root->parse, &context);
+
+ /* Assert we replaced all references */
+ Assert(context.refcount == 0);
+}
+
+static bool
+inline_cte_walker(Node *node, inline_cte_walker_context *context)
+{
+ if (node == NULL)
+ return false;
+ if (IsA(node, Query))
+ {
+ Query *query = (Query *) node;
+
+ context->levelsup++;
+
+ /*
+ * Visit the query's RTE nodes after their contents; otherwise
+ * query_tree_walker would descend into the newly inlined CTE query,
+ * which we don't want.
+ */
+ (void) query_tree_walker(query, inline_cte_walker, context,
+ QTW_EXAMINE_RTES_AFTER);
+
+ context->levelsup--;
+
+ return false;
+ }
+ else if (IsA(node, RangeTblEntry))
+ {
+ RangeTblEntry *rte = (RangeTblEntry *) node;
+
+ if (rte->rtekind == RTE_CTE &&
+ strcmp(rte->ctename, context->ctename) == 0 &&
+ rte->ctelevelsup == context->levelsup)
+ {
+ /*
+ * Found a reference to replace. Generate a copy of the CTE query
+ * with appropriate level adjustment for outer references (e.g.,
+ * to other CTEs).
+ */
+ Query *newquery = copyObject(context->ctequery);
+
+ if (context->levelsup > 0)
+ IncrementVarSublevelsUp((Node *) newquery, context->levelsup, 1);
+
+ /*
+ * Convert the RTE_CTE RTE into a RTE_SUBQUERY.
+ *
+ * Historically, a FOR UPDATE clause has been treated as extending
+ * into views and subqueries, but not into CTEs. We preserve this
+ * distinction by not trying to push rowmarks into the new
+ * subquery.
+ */
+ rte->rtekind = RTE_SUBQUERY;
+ rte->subquery = newquery;
+ rte->security_barrier = false;
+
+ /* Zero out CTE-specific fields */
+ rte->ctename = NULL;
+ rte->ctelevelsup = 0;
+ rte->self_reference = false;
+ rte->coltypes = NIL;
+ rte->coltypmods = NIL;
+ rte->colcollations = NIL;
+
+ /* Count the number of replacements we've done */
+ context->refcount--;
+ }
+
+ return false;
+ }
+
+ return expression_tree_walker(node, inline_cte_walker, context);
+}
+
+
+/*
* convert_ANY_sublink_to_join: try to convert an ANY SubLink to a join
*
* The caller has found an ANY SubLink at the top level of one of the query's
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index c1faf41..56c31d9 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -479,7 +479,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> row explicit_row implicit_row type_list array_expr_list
%type <node> case_expr case_arg when_clause case_default
%type <list> when_clause_list
-%type <ival> sub_type
+%type <ival> sub_type opt_materialized
%type <value> NumericOnly
%type <list> NumericOnly_list
%type <alias> alias_clause opt_alias_clause
@@ -11322,17 +11322,24 @@ cte_list:
| cte_list ',' common_table_expr { $$ = lappend($1, $3); }
;
-common_table_expr: name opt_name_list AS '(' PreparableStmt ')'
+common_table_expr: name opt_name_list AS opt_materialized '(' PreparableStmt ')'
{
CommonTableExpr *n = makeNode(CommonTableExpr);
n->ctename = $1;
n->aliascolnames = $2;
- n->ctequery = $5;
+ n->ctematerialized = $4;
+ n->ctequery = $6;
n->location = @1;
$$ = (Node *) n;
}
;
+opt_materialized:
+ MATERIALIZED { $$ = CTEMaterializeAlways; }
+ | NOT MATERIALIZED { $$ = CTEMaterializeNever; }
+ | /*EMPTY*/ { $$ = CTEMaterializeDefault; }
+ ;
+
opt_with_clause:
with_clause { $$ = $1; }
| /*EMPTY*/ { $$ = NULL; }
@@ -16214,6 +16221,7 @@ makeRecursiveViewSelect(char *relname, List *aliases, Node *query)
/* create common table expression */
cte->ctename = relname;
cte->aliascolnames = aliases;
+ cte->ctematerialized = CTEMaterializeDefault;
cte->ctequery = query;
cte->location = -1;
diff --git a/src/backend/utils/adt/ruleutils.c b/src/backend/utils/adt/ruleutils.c
index 302df16..1369602 100644
--- a/src/backend/utils/adt/ruleutils.c
+++ b/src/backend/utils/adt/ruleutils.c
@@ -5166,7 +5166,19 @@ get_with_clause(Query *query, deparse_context *context)
}
appendStringInfoChar(buf, ')');
}
- appendStringInfoString(buf, " AS (");
+ appendStringInfoString(buf, " AS ");
+ switch (cte->ctematerialized)
+ {
+ case CTEMaterializeDefault:
+ break;
+ case CTEMaterializeAlways:
+ appendStringInfoString(buf, "MATERIALIZED ");
+ break;
+ case CTEMaterializeNever:
+ appendStringInfoString(buf, "NOT MATERIALIZED ");
+ break;
+ }
+ appendStringInfoChar(buf, '(');
if (PRETTY_INDENT(context))
appendContextKeyword(context, "", 0, 0, 0);
get_query_def((Query *) cte->ctequery, buf, context->namespaces, NULL,
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index addc2c2..ce8161d 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -1399,11 +1399,19 @@ typedef struct OnConflictClause
*
* We don't currently support the SEARCH or CYCLE clause.
*/
+typedef enum CTEMaterialize
+{
+ CTEMaterializeDefault, /* no option specified */
+ CTEMaterializeAlways, /* MATERIALIZED */
+ CTEMaterializeNever /* NOT MATERIALIZED */
+} CTEMaterialize;
+
typedef struct CommonTableExpr
{
NodeTag type;
char *ctename; /* query name (never qualified) */
List *aliascolnames; /* optional list of column names */
+ CTEMaterialize ctematerialized; /* is this an optimization fence? */
/* SelectStmt/InsertStmt/etc before parse analysis, Query afterwards: */
Node *ctequery; /* the CTE's subquery */
int location; /* token location, or -1 if unknown */
diff --git a/src/test/regress/expected/rowsecurity.out b/src/test/regress/expected/rowsecurity.out
index 1d12b01..2e17049 100644
--- a/src/test/regress/expected/rowsecurity.out
+++ b/src/test/regress/expected/rowsecurity.out
@@ -2179,7 +2179,7 @@ EXPLAIN (COSTS OFF) EXECUTE plancache_test;
Filter: (((a % 2) = 0) AND f_leak(b))
(2 rows)
-PREPARE plancache_test2 AS WITH q AS (SELECT * FROM z1 WHERE f_leak(b)) SELECT * FROM q,z2;
+PREPARE plancache_test2 AS WITH q AS MATERIALIZED (SELECT * FROM z1 WHERE f_leak(b)) SELECT * FROM q,z2;
EXPLAIN (COSTS OFF) EXECUTE plancache_test2;
QUERY PLAN
-------------------------------------------------
@@ -2192,7 +2192,7 @@ EXPLAIN (COSTS OFF) EXECUTE plancache_test2;
-> Seq Scan on z2
(7 rows)
-PREPARE plancache_test3 AS WITH q AS (SELECT * FROM z2) SELECT * FROM q,z1 WHERE f_leak(z1.b);
+PREPARE plancache_test3 AS WITH q AS MATERIALIZED (SELECT * FROM z2) SELECT * FROM q,z1 WHERE f_leak(z1.b);
EXPLAIN (COSTS OFF) EXECUTE plancache_test3;
QUERY PLAN
-----------------------------------------------------
@@ -2826,7 +2826,7 @@ ALTER TABLE t1 ENABLE ROW LEVEL SECURITY;
GRANT ALL ON t1 TO regress_rls_bob;
INSERT INTO t1 (SELECT x, md5(x::text) FROM generate_series(0,20) x);
SET SESSION AUTHORIZATION regress_rls_bob;
-WITH cte1 AS (SELECT * FROM t1 WHERE f_leak(b)) SELECT * FROM cte1;
+WITH cte1 AS MATERIALIZED (SELECT * FROM t1 WHERE f_leak(b)) SELECT * FROM cte1;
NOTICE: f_leak => cfcd208495d565ef66e7dff9f98764da
NOTICE: f_leak => c81e728d9d4c2f636f067f89cc14862c
NOTICE: f_leak => a87ff679a2f3e71d9181a67b7542122c
@@ -2853,7 +2853,8 @@ NOTICE: f_leak => 98f13708210194c475687be6106a3b84
20 | 98f13708210194c475687be6106a3b84
(11 rows)
-EXPLAIN (COSTS OFF) WITH cte1 AS (SELECT * FROM t1 WHERE f_leak(b)) SELECT * FROM cte1;
+EXPLAIN (COSTS OFF)
+WITH cte1 AS MATERIALIZED (SELECT * FROM t1 WHERE f_leak(b)) SELECT * FROM cte1;
QUERY PLAN
-------------------------------------------------
CTE Scan on cte1
diff --git a/src/test/regress/expected/rowtypes.out b/src/test/regress/expected/rowtypes.out
index d6a1a33..14757b4 100644
--- a/src/test/regress/expected/rowtypes.out
+++ b/src/test/regress/expected/rowtypes.out
@@ -1037,7 +1037,7 @@ from (values (1,row(1,2)), (1,row(null,null)), (1,null),
(6 rows)
explain (verbose, costs off)
-with r(a,b) as
+with r(a,b) as materialized
(values (1,row(1,2)), (1,row(null,null)), (1,null),
(null,row(1,2)), (null,row(null,null)), (null,null) )
select r, r is null as isnull, r is not null as isnotnull from r;
@@ -1050,7 +1050,7 @@ select r, r is null as isnull, r is not null as isnotnull from r;
Output: "*VALUES*".column1, "*VALUES*".column2
(5 rows)
-with r(a,b) as
+with r(a,b) as materialized
(values (1,row(1,2)), (1,row(null,null)), (1,null),
(null,row(1,2)), (null,row(null,null)), (null,null) )
select r, r is null as isnull, r is not null as isnotnull from r;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index e384cd2..bb10e27 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -3108,7 +3108,7 @@ explain (costs off) INSERT INTO hats VALUES ('h8', 'forbidden') RETURNING *;
(5 rows)
-- ensure upserting into a rule, with a CTE (different offsets!) works
-WITH data(hat_name, hat_color) AS (
+WITH data(hat_name, hat_color) AS MATERIALIZED (
VALUES ('h8', 'green'),
('h9', 'blue'),
('h7', 'forbidden')
@@ -3122,7 +3122,8 @@ RETURNING *;
h9 | blue
(2 rows)
-EXPLAIN (costs off) WITH data(hat_name, hat_color) AS (
+EXPLAIN (costs off)
+WITH data(hat_name, hat_color) AS MATERIALIZED (
VALUES ('h8', 'green'),
('h9', 'blue'),
('h7', 'forbidden')
diff --git a/src/test/regress/expected/subselect.out b/src/test/regress/expected/subselect.out
index 588d069..98feb41 100644
--- a/src/test/regress/expected/subselect.out
+++ b/src/test/regress/expected/subselect.out
@@ -1154,3 +1154,159 @@ fetch backward all in c1;
(2 rows)
commit;
+--
+-- Tests for CTE inlining behavior
+--
+-- Basic subquery that can be inlined
+explain (verbose, costs off)
+with x as (select * from (select f1 from subselect_tbl) ss)
+select * from x where f1 = 1;
+ QUERY PLAN
+----------------------------------
+ Seq Scan on public.subselect_tbl
+ Output: subselect_tbl.f1
+ Filter: (subselect_tbl.f1 = 1)
+(3 rows)
+
+-- Explicitly request materialization
+explain (verbose, costs off)
+with x as materialized (select * from (select f1 from subselect_tbl) ss)
+select * from x where f1 = 1;
+ QUERY PLAN
+------------------------------------------
+ CTE Scan on x
+ Output: x.f1
+ Filter: (x.f1 = 1)
+ CTE x
+ -> Seq Scan on public.subselect_tbl
+ Output: subselect_tbl.f1
+(6 rows)
+
+-- Stable functions are safe to inline
+explain (verbose, costs off)
+with x as (select * from (select f1, now() from subselect_tbl) ss)
+select * from x where f1 = 1;
+ QUERY PLAN
+-----------------------------------
+ Seq Scan on public.subselect_tbl
+ Output: subselect_tbl.f1, now()
+ Filter: (subselect_tbl.f1 = 1)
+(3 rows)
+
+-- Volatile functions prevent inlining
+explain (verbose, costs off)
+with x as (select * from (select f1, random() from subselect_tbl) ss)
+select * from x where f1 = 1;
+ QUERY PLAN
+----------------------------------------------
+ CTE Scan on x
+ Output: x.f1, x.random
+ Filter: (x.f1 = 1)
+ CTE x
+ -> Seq Scan on public.subselect_tbl
+ Output: subselect_tbl.f1, random()
+(6 rows)
+
+-- SELECT FOR UPDATE cannot be inlined
+explain (verbose, costs off)
+with x as (select * from (select f1 from subselect_tbl for update) ss)
+select * from x where f1 = 1;
+ QUERY PLAN
+--------------------------------------------------------------------
+ CTE Scan on x
+ Output: x.f1
+ Filter: (x.f1 = 1)
+ CTE x
+ -> Subquery Scan on ss
+ Output: ss.f1
+ -> LockRows
+ Output: subselect_tbl.f1, subselect_tbl.ctid
+ -> Seq Scan on public.subselect_tbl
+ Output: subselect_tbl.f1, subselect_tbl.ctid
+(10 rows)
+
+-- Multiply-referenced CTEs are inlined only when requested
+explain (verbose, costs off)
+with x as (select * from (select f1, now() as n from subselect_tbl) ss)
+select * from x, x x2 where x.n = x2.n;
+ QUERY PLAN
+-------------------------------------------
+ Merge Join
+ Output: x.f1, x.n, x2.f1, x2.n
+ Merge Cond: (x.n = x2.n)
+ CTE x
+ -> Seq Scan on public.subselect_tbl
+ Output: subselect_tbl.f1, now()
+ -> Sort
+ Output: x.f1, x.n
+ Sort Key: x.n
+ -> CTE Scan on x
+ Output: x.f1, x.n
+ -> Sort
+ Output: x2.f1, x2.n
+ Sort Key: x2.n
+ -> CTE Scan on x x2
+ Output: x2.f1, x2.n
+(16 rows)
+
+explain (verbose, costs off)
+with x as not materialized (select * from (select f1, now() as n from subselect_tbl) ss)
+select * from x, x x2 where x.n = x2.n;
+ QUERY PLAN
+----------------------------------------------------------------------------
+ Result
+ Output: subselect_tbl.f1, now(), subselect_tbl_1.f1, now()
+ One-Time Filter: (now() = now())
+ -> Nested Loop
+ Output: subselect_tbl.f1, subselect_tbl_1.f1
+ -> Seq Scan on public.subselect_tbl
+ Output: subselect_tbl.f1, subselect_tbl.f2, subselect_tbl.f3
+ -> Materialize
+ Output: subselect_tbl_1.f1
+ -> Seq Scan on public.subselect_tbl subselect_tbl_1
+ Output: subselect_tbl_1.f1
+(11 rows)
+
+-- Check handling of outer references
+explain (verbose, costs off)
+with x as (select * from int4_tbl)
+select * from (with y as (select * from x) select * from y) ss;
+ QUERY PLAN
+-----------------------------
+ Seq Scan on public.int4_tbl
+ Output: int4_tbl.f1
+(2 rows)
+
+explain (verbose, costs off)
+with x as materialized (select * from int4_tbl)
+select * from (with y as (select * from x) select * from y) ss;
+ QUERY PLAN
+-------------------------------------
+ CTE Scan on x
+ Output: x.f1
+ CTE x
+ -> Seq Scan on public.int4_tbl
+ Output: int4_tbl.f1
+(5 rows)
+
+-- Ensure that we inline the currect CTE when there are
+-- multiple CTEs with the same name
+explain (verbose, costs off)
+with x as (select 1 as y)
+select * from (with x as (select 2 as y) select * from x) ss;
+ QUERY PLAN
+-------------
+ Result
+ Output: 2
+(2 rows)
+
+-- Row marks are not pushed into CTEs
+explain (verbose, costs off)
+with x as (select * from subselect_tbl)
+select * from x for update;
+ QUERY PLAN
+----------------------------------------------------------------
+ Seq Scan on public.subselect_tbl
+ Output: subselect_tbl.f1, subselect_tbl.f2, subselect_tbl.f3
+(2 rows)
+
diff --git a/src/test/regress/sql/rowsecurity.sql b/src/test/regress/sql/rowsecurity.sql
index 38e9b38..52da276 100644
--- a/src/test/regress/sql/rowsecurity.sql
+++ b/src/test/regress/sql/rowsecurity.sql
@@ -840,10 +840,10 @@ EXPLAIN (COSTS OFF) SELECT * FROM z1 WHERE f_leak(b);
PREPARE plancache_test AS SELECT * FROM z1 WHERE f_leak(b);
EXPLAIN (COSTS OFF) EXECUTE plancache_test;
-PREPARE plancache_test2 AS WITH q AS (SELECT * FROM z1 WHERE f_leak(b)) SELECT * FROM q,z2;
+PREPARE plancache_test2 AS WITH q AS MATERIALIZED (SELECT * FROM z1 WHERE f_leak(b)) SELECT * FROM q,z2;
EXPLAIN (COSTS OFF) EXECUTE plancache_test2;
-PREPARE plancache_test3 AS WITH q AS (SELECT * FROM z2) SELECT * FROM q,z1 WHERE f_leak(z1.b);
+PREPARE plancache_test3 AS WITH q AS MATERIALIZED (SELECT * FROM z2) SELECT * FROM q,z1 WHERE f_leak(z1.b);
EXPLAIN (COSTS OFF) EXECUTE plancache_test3;
SET ROLE regress_rls_group1;
@@ -1071,8 +1071,9 @@ INSERT INTO t1 (SELECT x, md5(x::text) FROM generate_series(0,20) x);
SET SESSION AUTHORIZATION regress_rls_bob;
-WITH cte1 AS (SELECT * FROM t1 WHERE f_leak(b)) SELECT * FROM cte1;
-EXPLAIN (COSTS OFF) WITH cte1 AS (SELECT * FROM t1 WHERE f_leak(b)) SELECT * FROM cte1;
+WITH cte1 AS MATERIALIZED (SELECT * FROM t1 WHERE f_leak(b)) SELECT * FROM cte1;
+EXPLAIN (COSTS OFF)
+WITH cte1 AS MATERIALIZED (SELECT * FROM t1 WHERE f_leak(b)) SELECT * FROM cte1;
WITH cte1 AS (UPDATE t1 SET a = a + 1 RETURNING *) SELECT * FROM cte1; --fail
WITH cte1 AS (UPDATE t1 SET a = a RETURNING *) SELECT * FROM cte1; --ok
diff --git a/src/test/regress/sql/rowtypes.sql b/src/test/regress/sql/rowtypes.sql
index e6d3898..b3b0774 100644
--- a/src/test/regress/sql/rowtypes.sql
+++ b/src/test/regress/sql/rowtypes.sql
@@ -428,12 +428,12 @@ from (values (1,row(1,2)), (1,row(null,null)), (1,null),
(null,row(1,2)), (null,row(null,null)), (null,null) ) r(a,b);
explain (verbose, costs off)
-with r(a,b) as
+with r(a,b) as materialized
(values (1,row(1,2)), (1,row(null,null)), (1,null),
(null,row(1,2)), (null,row(null,null)), (null,null) )
select r, r is null as isnull, r is not null as isnotnull from r;
-with r(a,b) as
+with r(a,b) as materialized
(values (1,row(1,2)), (1,row(null,null)), (1,null),
(null,row(1,2)), (null,row(null,null)), (null,null) )
select r, r is null as isnull, r is not null as isnotnull from r;
diff --git a/src/test/regress/sql/rules.sql b/src/test/regress/sql/rules.sql
index f4ee30e..6666951 100644
--- a/src/test/regress/sql/rules.sql
+++ b/src/test/regress/sql/rules.sql
@@ -1132,7 +1132,7 @@ SELECT tablename, rulename, definition FROM pg_rules
explain (costs off) INSERT INTO hats VALUES ('h8', 'forbidden') RETURNING *;
-- ensure upserting into a rule, with a CTE (different offsets!) works
-WITH data(hat_name, hat_color) AS (
+WITH data(hat_name, hat_color) AS MATERIALIZED (
VALUES ('h8', 'green'),
('h9', 'blue'),
('h7', 'forbidden')
@@ -1140,7 +1140,8 @@ WITH data(hat_name, hat_color) AS (
INSERT INTO hats
SELECT * FROM data
RETURNING *;
-EXPLAIN (costs off) WITH data(hat_name, hat_color) AS (
+EXPLAIN (costs off)
+WITH data(hat_name, hat_color) AS MATERIALIZED (
VALUES ('h8', 'green'),
('h9', 'blue'),
('h7', 'forbidden')
diff --git a/src/test/regress/sql/subselect.sql b/src/test/regress/sql/subselect.sql
index 843f511..f4093e1 100644
--- a/src/test/regress/sql/subselect.sql
+++ b/src/test/regress/sql/subselect.sql
@@ -625,3 +625,61 @@ move forward all in c1;
fetch backward all in c1;
commit;
+
+--
+-- Tests for CTE inlining behavior
+--
+
+-- Basic subquery that can be inlined
+explain (verbose, costs off)
+with x as (select * from (select f1 from subselect_tbl) ss)
+select * from x where f1 = 1;
+
+-- Explicitly request materialization
+explain (verbose, costs off)
+with x as materialized (select * from (select f1 from subselect_tbl) ss)
+select * from x where f1 = 1;
+
+-- Stable functions are safe to inline
+explain (verbose, costs off)
+with x as (select * from (select f1, now() from subselect_tbl) ss)
+select * from x where f1 = 1;
+
+-- Volatile functions prevent inlining
+explain (verbose, costs off)
+with x as (select * from (select f1, random() from subselect_tbl) ss)
+select * from x where f1 = 1;
+
+-- SELECT FOR UPDATE cannot be inlined
+explain (verbose, costs off)
+with x as (select * from (select f1 from subselect_tbl for update) ss)
+select * from x where f1 = 1;
+
+-- Multiply-referenced CTEs are inlined only when requested
+explain (verbose, costs off)
+with x as (select * from (select f1, now() as n from subselect_tbl) ss)
+select * from x, x x2 where x.n = x2.n;
+
+explain (verbose, costs off)
+with x as not materialized (select * from (select f1, now() as n from subselect_tbl) ss)
+select * from x, x x2 where x.n = x2.n;
+
+-- Check handling of outer references
+explain (verbose, costs off)
+with x as (select * from int4_tbl)
+select * from (with y as (select * from x) select * from y) ss;
+
+explain (verbose, costs off)
+with x as materialized (select * from int4_tbl)
+select * from (with y as (select * from x) select * from y) ss;
+
+-- Ensure that we inline the currect CTE when there are
+-- multiple CTEs with the same name
+explain (verbose, costs off)
+with x as (select 1 as y)
+select * from (with x as (select 2 as y) select * from x) ss;
+
+-- Row marks are not pushed into CTEs
+explain (verbose, costs off)
+with x as (select * from subselect_tbl)
+select * from x for update;
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-22 03:47 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-25 22:22 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-26 22:55 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
@ 2019-01-27 13:22 ` Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-27 15:21 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 19:01 ` Re: Early WIP/PoC for inlining CTEs Robert Haas <robertmhaas@gmail.com>
1 sibling, 2 replies; 112+ messages in thread
From: Andrew Gierth @ 2019-01-27 13:22 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Andreas Karlsson <andreas@proxel.se>; David Fetter <david@fetter.org>; Thomas Munro <thomas.munro@enterprisedb.com>; pgsql-hackers
>>>>> "Tom" == Tom Lane <tgl@sss.pgh.pa.us> writes:
Tom> I was interested to find, while writing the docs, that it's a real
Tom> struggle to invent plausible reasons to write MATERIALIZED given
Tom> the above specification. You pretty much have to have lied to the
Tom> planner, eg by making a volatile function that's not marked
Tom> volatile, before there's a real need for that. Am I missing
Tom> something? If I'm not, then we're in a really good place
Tom> backwards-compatibility-wise, because the new default behavior
Tom> shouldn't break any cases where people weren't cheating.
The cases where the new default will break things are where people used
WITH to force a choice of join order or otherwise constrain the planner
in order to avoid a misplan.
I'm not sure we should nail down the rule that the absence of NOT
MATERIALIZED will mean a multiply-referenced CTE is evaluated once. One
would hope that in the future the planner might be taught to inline or
not in that case depending on cost. I think it makes more sense to say
that we never inline if MATERIALIZED is specified, that we always inline
if NOT MATERIALIZED is specified, and that if neither is specified the
planner will choose (but perhaps note that currently it always chooses
only based on refcount).
--
Andrew (irc:RhodiumToad)
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-22 03:47 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-25 22:22 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-26 22:55 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-27 13:22 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
@ 2019-01-27 15:21 ` Tom Lane <tgl@sss.pgh.pa.us>
2019-01-27 17:24 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
1 sibling, 1 reply; 112+ messages in thread
From: Tom Lane @ 2019-01-27 15:21 UTC (permalink / raw)
To: Andrew Gierth <andrew@tao11.riddles.org.uk>; +Cc: Andreas Karlsson <andreas@proxel.se>; David Fetter <david@fetter.org>; Thomas Munro <thomas.munro@enterprisedb.com>; pgsql-hackers
Andrew Gierth <andrew@tao11.riddles.org.uk> writes:
> I'm not sure we should nail down the rule that the absence of NOT
> MATERIALIZED will mean a multiply-referenced CTE is evaluated once. One
> would hope that in the future the planner might be taught to inline or
> not in that case depending on cost. I think it makes more sense to say
> that we never inline if MATERIALIZED is specified, that we always inline
> if NOT MATERIALIZED is specified, and that if neither is specified the
> planner will choose (but perhaps note that currently it always chooses
> only based on refcount).
I have no objection to documenting it like that; I just don't want us
to go off into the weeds trying to actually implement something smarter
for v12.
regards, tom lane
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-22 03:47 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-25 22:22 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-26 22:55 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-27 13:22 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-27 15:21 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
@ 2019-01-27 17:24 ` Andreas Karlsson <andreas@proxel.se>
0 siblings, 0 replies; 112+ messages in thread
From: Andreas Karlsson @ 2019-01-27 17:24 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; Andrew Gierth <andrew@tao11.riddles.org.uk>; +Cc: David Fetter <david@fetter.org>; Thomas Munro <thomas.munro@enterprisedb.com>; pgsql-hackers
On 1/27/19 4:21 PM, Tom Lane wrote:
> Andrew Gierth <andrew@tao11.riddles.org.uk> writes:
>> I'm not sure we should nail down the rule that the absence of NOT
>> MATERIALIZED will mean a multiply-referenced CTE is evaluated once. One
>> would hope that in the future the planner might be taught to inline or
>> not in that case depending on cost. I think it makes more sense to say
>> that we never inline if MATERIALIZED is specified, that we always inline
>> if NOT MATERIALIZED is specified, and that if neither is specified the
>> planner will choose (but perhaps note that currently it always chooses
>> only based on refcount).
>
> I have no objection to documenting it like that; I just don't want us
> to go off into the weeds trying to actually implement something smarter
> for v12.
+1
Andreas
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-22 03:47 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-25 22:22 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-26 22:55 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-27 13:22 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
@ 2019-01-28 19:01 ` Robert Haas <robertmhaas@gmail.com>
1 sibling, 0 replies; 112+ messages in thread
From: Robert Haas @ 2019-01-28 19:01 UTC (permalink / raw)
To: Andrew Gierth <andrew@tao11.riddles.org.uk>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Andreas Karlsson <andreas@proxel.se>; David Fetter <david@fetter.org>; Thomas Munro <thomas.munro@enterprisedb.com>; pgsql-hackers
On Sun, Jan 27, 2019 at 8:23 AM Andrew Gierth
<andrew@tao11.riddles.org.uk> wrote:
> I think it makes more sense to say
> that we never inline if MATERIALIZED is specified, that we always inline
> if NOT MATERIALIZED is specified, and that if neither is specified the
> planner will choose (but perhaps note that currently it always chooses
> only based on refcount).
I, too, like this approach.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-22 03:47 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-25 22:22 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2019-01-26 22:55 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
@ 2019-01-27 18:38 ` Andreas Karlsson <andreas@proxel.se>
1 sibling, 0 replies; 112+ messages in thread
From: Andreas Karlsson @ 2019-01-27 18:38 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Andrew Gierth <andrew@tao11.riddles.org.uk>; David Fetter <david@fetter.org>; Thomas Munro <thomas.munro@enterprisedb.com>; pgsql-hackers
On 1/26/19 11:55 PM, Tom Lane wrote:> Hearing no immediate pushback on
that proposal, I went ahead and made
> a version of the patch that does it like that, as attached. I also took
> a stab at documenting it fully.
Thanks! This version of the patch looks solid, including the
documentation. The only remaining question I see is the one Andrew
raised about if we should change the language to allow for future
versions of PostgreSQL to add costing for when the same CTE is
referenced multiple times.
Andreas
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 02:55 ` Re: Early WIP/PoC for inlining CTEs Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-10-01 12:55 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2018-11-16 21:15 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
2018-12-31 23:46 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-10 01:28 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-22 03:47 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
2019-01-25 22:22 ` Re: Early WIP/PoC for inlining CTEs Tom Lane <tgl@sss.pgh.pa.us>
@ 2019-01-27 00:15 ` Marko Tiikkaja <marko@joh.to>
1 sibling, 0 replies; 112+ messages in thread
From: Marko Tiikkaja @ 2019-01-27 00:15 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Andreas Karlsson <andreas@proxel.se>; Andrew Gierth <andrew@tao11.riddles.org.uk>; David Fetter <david@fetter.org>; Thomas Munro <thomas.munro@enterprisedb.com>; pgsql-hackers
On Sat, Jan 26, 2019 at 12:22 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Therefore, I'm reversing my previous opinion that we should not have
> an explicit NOT MATERIALIZED option. I think we should add that, and
> the behavior ought to be:
>
> * No option given: inline if there's exactly one reference.
>
> * With MATERIALIZED: never inline.
>
> * With NOT MATERIALIZED: inline regardless of the number of references.
>
This much has been obvious to most people for a long time.
.m
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
@ 2018-07-25 15:32 ` Nico Williams <nico@cryptonector.com>
2018-07-25 16:08 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
1 sibling, 1 reply; 112+ messages in thread
From: Nico Williams @ 2018-07-25 15:32 UTC (permalink / raw)
To: David Fetter <david@fetter.org>; +Cc: Andrew Gierth <andrew@tao11.riddles.org.uk>; pgsql-hackers
On Wed, Jul 25, 2018 at 07:42:37AM +0200, David Fetter wrote:
> Please find attached a version rebased atop 167075be3ab1547e18 with
> what I believe are appropriate changes to regression test output. The
> other changes to the regression tests output are somewhat puzzling, as
> they change the actual results of queries. I've also attached both
> the "leftover" diff and the files to which it should be applied.
I think the SQL programmer needs some control over whether a CTE is:
- a materialized view -- and therefore a barrier
- a view (which can then be inlined by the optimizer)
It is possible to add a keyword for this purpose in the WITH syntax:
WITH VIEW (...) AS a_view
, MATERIALIZED VIEW (...) AS a_barrier
...;
This would be a lot like creating TEMP views, but without the catalog
overhead.
(I wonder how hard it would be to partiion the OID namespace into
temp/persistent ranges so that temp schema elements need not be written
into the catalog.)
Nico
--
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:32 ` Re: Early WIP/PoC for inlining CTEs Nico Williams <nico@cryptonector.com>
@ 2018-07-25 16:08 ` Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 18:36 ` Re: Early WIP/PoC for inlining CTEs Nico Williams <nico@cryptonector.com>
2018-07-26 12:51 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
0 siblings, 2 replies; 112+ messages in thread
From: Andrew Gierth @ 2018-07-25 16:08 UTC (permalink / raw)
To: Nico Williams <nico@cryptonector.com>; +Cc: David Fetter <david@fetter.org>; pgsql-hackers
>>>>> "Nico" == Nico Williams <nico@cryptonector.com> writes:
Nico> It is possible to add a keyword for this purpose in the WITH syntax:
Nico> WITH VIEW (...) AS a_view
The existing (and standard) syntax is WITH ctename AS (query).
Syntaxes that have been suggested for explicitly controlling the
materialization are along the lines of:
WITH ctename AS [[NOT] MATERIALIZED] (query)
(MATERIALIZED is already an un-reserved keyword)
--
Andrew (irc:RhodiumToad)
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:32 ` Re: Early WIP/PoC for inlining CTEs Nico Williams <nico@cryptonector.com>
2018-07-25 16:08 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
@ 2018-07-25 18:36 ` Nico Williams <nico@cryptonector.com>
1 sibling, 0 replies; 112+ messages in thread
From: Nico Williams @ 2018-07-25 18:36 UTC (permalink / raw)
To: Andrew Gierth <andrew@tao11.riddles.org.uk>; +Cc: David Fetter <david@fetter.org>; pgsql-hackers
On Wed, Jul 25, 2018 at 05:08:43PM +0100, Andrew Gierth wrote:
> Nico> It is possible to add a keyword for this purpose in the WITH syntax:
>
> Nico> WITH VIEW (...) AS a_view
>
> The existing (and standard) syntax is WITH ctename AS (query).
Oy, I flubbed that up.
> Syntaxes that have been suggested for explicitly controlling the
> materialization are along the lines of:
>
> WITH ctename AS [[NOT] MATERIALIZED] (query)
>
> (MATERIALIZED is already an un-reserved keyword)
Works for me.
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:32 ` Re: Early WIP/PoC for inlining CTEs Nico Williams <nico@cryptonector.com>
2018-07-25 16:08 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
@ 2018-07-26 12:51 ` Andreas Karlsson <andreas@proxel.se>
2018-07-26 14:46 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-27 16:33 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
1 sibling, 2 replies; 112+ messages in thread
From: Andreas Karlsson @ 2018-07-26 12:51 UTC (permalink / raw)
To: Andrew Gierth <andrew@tao11.riddles.org.uk>; Nico Williams <nico@cryptonector.com>; +Cc: David Fetter <david@fetter.org>; pgsql-hackers
On 07/25/2018 06:08 PM, Andrew Gierth wrote:
> WITH ctename AS [[NOT] MATERIALIZED] (query)
I think "NOT MATERIALIZED" would be a bit misleading since the planner
may choose to materialize anyway, so I suggest skipping that part of the
syntax unless there is a really strong reason for having it.
Andreas
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:32 ` Re: Early WIP/PoC for inlining CTEs Nico Williams <nico@cryptonector.com>
2018-07-25 16:08 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-26 12:51 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
@ 2018-07-26 14:46 ` David Fetter <david@fetter.org>
1 sibling, 0 replies; 112+ messages in thread
From: David Fetter @ 2018-07-26 14:46 UTC (permalink / raw)
To: Andreas Karlsson <andreas@proxel.se>; +Cc: Andrew Gierth <andrew@tao11.riddles.org.uk>; Nico Williams <nico@cryptonector.com>; pgsql-hackers
On Thu, Jul 26, 2018 at 02:51:53PM +0200, Andreas Karlsson wrote:
> On 07/25/2018 06:08 PM, Andrew Gierth wrote:
> >WITH ctename AS [[NOT] MATERIALIZED] (query)
>
> I think "NOT MATERIALIZED" would be a bit misleading since the
> planner may choose to materialize anyway, so I suggest skipping that
> part of the syntax unless there is a really strong reason for having
> it.
If we're going to give people specific knobs to turn as part of
queries to affect query plans[1], we should think it through at a much
higher level than this.
If we're not going to do that right away, we just need to treat
instances where the wrong call was made as planner bugs and address
them at that level.
Best,
David.
[1] Not to use the word that starts with 'h' and sounds like lints http://pghintplan.osdn.jp/
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778
Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate
^ permalink raw reply [nested|flat] 112+ messages in thread
* Re: Early WIP/PoC for inlining CTEs
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 05:42 ` Re: Early WIP/PoC for inlining CTEs David Fetter <david@fetter.org>
2018-07-25 15:32 ` Re: Early WIP/PoC for inlining CTEs Nico Williams <nico@cryptonector.com>
2018-07-25 16:08 ` Re: Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-26 12:51 ` Re: Early WIP/PoC for inlining CTEs Andreas Karlsson <andreas@proxel.se>
@ 2018-07-27 16:33 ` Andrew Gierth <andrew@tao11.riddles.org.uk>
1 sibling, 0 replies; 112+ messages in thread
From: Andrew Gierth @ 2018-07-27 16:33 UTC (permalink / raw)
To: Andreas Karlsson <andreas@proxel.se>; +Cc: Nico Williams <nico@cryptonector.com>; David Fetter <david@fetter.org>; pgsql-hackers
>>>>> "Andreas" == Andreas Karlsson <andreas@proxel.se> writes:
>> WITH ctename AS [[NOT] MATERIALIZED] (query)
Andreas> I think "NOT MATERIALIZED" would be a bit misleading since the
Andreas> planner may choose to materialize anyway,
It would certainly be possible to make an explicit NOT MATERIALIZED
override such a choice by the planner, or produce an error if it is
actually impossible to do so (e.g. using NOT MATERIALIZED on a wCTE)
--
Andrew (irc:RhodiumToad)
^ permalink raw reply [nested|flat] 112+ messages in thread
end of thread, other threads:[~2019-04-09 19:50 UTC | newest]
Thread overview: 112+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2018-07-24 22:28 Early WIP/PoC for inlining CTEs Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-24 23:03 ` Jeremy Finzel <finzelj@gmail.com>
2018-07-24 23:10 ` Andres Freund <andres@anarazel.de>
2018-07-24 23:48 ` Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-24 23:49 ` Tom Lane <tgl@sss.pgh.pa.us>
2018-07-24 23:54 ` Andres Freund <andres@anarazel.de>
2018-07-24 23:57 ` Tom Lane <tgl@sss.pgh.pa.us>
2018-07-25 00:04 ` Andres Freund <andres@anarazel.de>
2018-07-25 00:08 ` Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 00:27 ` Andres Freund <andres@anarazel.de>
2018-07-25 18:39 ` Nico Williams <nico@cryptonector.com>
2018-07-25 00:04 ` Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-24 23:50 ` Gavin Flower <GavinFlower@archidevsys.co.nz>
2019-01-11 21:25 ` Merlin Moncure <mmoncure@gmail.com>
2018-07-25 05:42 ` David Fetter <david@fetter.org>
2018-07-25 15:18 ` Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 19:14 ` David Fetter <david@fetter.org>
2018-07-27 02:55 ` Thomas Munro <thomas.munro@enterprisedb.com>
2018-07-27 08:10 ` David Fetter <david@fetter.org>
2018-08-08 04:55 ` Thomas Munro <thomas.munro@enterprisedb.com>
2018-08-08 05:02 ` Andres Freund <andres@anarazel.de>
2019-03-13 00:45 ` Tatsuo Ishii <ishii@sraoss.co.jp>
2018-10-01 12:55 ` Andreas Karlsson <andreas@proxel.se>
2018-10-02 10:03 ` Andreas Karlsson <andreas@proxel.se>
2018-10-03 15:57 ` David Fetter <david@fetter.org>
2018-10-04 09:22 ` Andreas Karlsson <andreas@proxel.se>
2018-10-04 18:49 ` David Fetter <david@fetter.org>
2018-10-05 00:40 ` Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-10-05 22:43 ` David Fetter <david@fetter.org>
2018-10-06 02:48 ` Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-11-16 21:15 ` Tom Lane <tgl@sss.pgh.pa.us>
2018-11-16 22:41 ` David Fetter <david@fetter.org>
2018-11-16 23:04 ` Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-11-16 23:24 ` Tom Lane <tgl@sss.pgh.pa.us>
2018-11-17 02:12 ` Stephen Frost <sfrost@snowman.net>
2018-11-19 03:38 ` Craig Ringer <craig@2ndquadrant.com>
2018-12-31 23:46 ` Andreas Karlsson <andreas@proxel.se>
2019-01-01 00:42 ` Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-02 04:31 ` Andreas Karlsson <andreas@proxel.se>
2019-01-01 02:18 ` Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-02 04:29 ` Andreas Karlsson <andreas@proxel.se>
2019-01-10 01:28 ` Andreas Karlsson <andreas@proxel.se>
2019-01-10 02:24 ` Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 16:12 ` Robert Haas <robertmhaas@gmail.com>
2019-01-11 18:19 ` Andres Freund <andres@anarazel.de>
2019-01-11 18:58 ` Thomas Munro <thomas.munro@enterprisedb.com>
2019-01-11 19:04 ` Andres Freund <andres@anarazel.de>
2019-01-11 19:10 ` Robert Haas <robertmhaas@gmail.com>
2019-01-11 19:39 ` Mike Rylander <mrylander@gmail.com>
2019-01-12 05:51 ` Pavel Stehule <pavel.stehule@gmail.com>
2019-01-17 15:48 ` Andreas Karlsson <andreas@proxel.se>
2019-01-17 15:52 ` Tom Lane <tgl@sss.pgh.pa.us>
2019-01-18 20:34 ` Robert Haas <robertmhaas@gmail.com>
2019-01-18 20:42 ` Andres Freund <andres@anarazel.de>
2019-01-18 20:44 ` Robert Haas <robertmhaas@gmail.com>
2019-01-21 13:40 ` Andreas Karlsson <andreas@proxel.se>
2019-01-21 15:27 ` Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 19:00 ` Robert Haas <robertmhaas@gmail.com>
2019-01-28 20:35 ` Tom Lane <tgl@sss.pgh.pa.us>
2019-01-28 21:54 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-01-29 02:36 ` Andreas Karlsson <andreas@proxel.se>
2019-01-28 22:05 ` Tom Lane <tgl@sss.pgh.pa.us>
2019-01-29 04:51 ` Michael Paquier <michael@paquier.xyz>
2019-01-29 19:52 ` Tom Lane <tgl@sss.pgh.pa.us>
2019-01-29 20:36 ` Merlin Moncure <mmoncure@gmail.com>
2019-01-29 20:46 ` Tom Lane <tgl@sss.pgh.pa.us>
2019-01-30 04:56 ` Merlin Moncure <mmoncure@gmail.com>
2019-01-29 20:56 ` Chapman Flack <chap@anastigmatix.net>
2019-01-30 02:35 ` David Fetter <david@fetter.org>
2019-01-30 03:42 ` Tom Lane <tgl@sss.pgh.pa.us>
2019-02-02 19:01 ` Tom Lane <tgl@sss.pgh.pa.us>
2019-02-03 08:14 ` Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-02-09 20:52 ` Tom Lane <tgl@sss.pgh.pa.us>
2019-02-14 15:11 ` Tom Lane <tgl@sss.pgh.pa.us>
2019-02-14 15:22 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-02-14 16:02 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2019-02-14 19:22 ` Merlin Moncure <mmoncure@gmail.com>
2019-02-15 22:36 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-02-16 21:14 ` Tom Lane <tgl@sss.pgh.pa.us>
2019-02-16 21:33 ` Jonathan S. Katz <jkatz@postgresql.org>
2019-02-17 00:34 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-02-17 02:52 ` Andreas Karlsson <andreas@proxel.se>
2019-02-26 11:41 ` Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-02-26 14:51 ` Tom Lane <tgl@sss.pgh.pa.us>
2019-02-27 01:11 ` Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-02-27 04:45 ` Tom Lane <tgl@sss.pgh.pa.us>
2019-02-27 05:16 ` Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-04-09 19:50 ` Tom Lane <tgl@sss.pgh.pa.us>
2019-02-15 02:34 ` Andreas Karlsson <andreas@proxel.se>
2019-02-06 09:00 ` Bruce Momjian <bruce@momjian.us>
2019-02-14 15:25 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-02-15 01:15 ` Bruce Momjian <bruce@momjian.us>
2019-02-03 11:16 ` Vik Fearing <vik.fearing@2ndquadrant.com>
2019-02-03 15:39 ` Tom Lane <tgl@sss.pgh.pa.us>
2019-01-22 05:13 ` Gavin Flower <GavinFlower@archidevsys.co.nz>
2019-01-11 18:49 ` David Fetter <david@fetter.org>
2019-01-11 18:58 ` Robert Haas <robertmhaas@gmail.com>
2019-01-22 03:47 ` Andreas Karlsson <andreas@proxel.se>
2019-01-25 22:22 ` Tom Lane <tgl@sss.pgh.pa.us>
2019-01-26 22:55 ` Tom Lane <tgl@sss.pgh.pa.us>
2019-01-27 13:22 ` Andrew Gierth <andrew@tao11.riddles.org.uk>
2019-01-27 15:21 ` Tom Lane <tgl@sss.pgh.pa.us>
2019-01-27 17:24 ` Andreas Karlsson <andreas@proxel.se>
2019-01-28 19:01 ` Robert Haas <robertmhaas@gmail.com>
2019-01-27 18:38 ` Andreas Karlsson <andreas@proxel.se>
2019-01-27 00:15 ` Marko Tiikkaja <marko@joh.to>
2018-07-25 15:32 ` Nico Williams <nico@cryptonector.com>
2018-07-25 16:08 ` Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-07-25 18:36 ` Nico Williams <nico@cryptonector.com>
2018-07-26 12:51 ` Andreas Karlsson <andreas@proxel.se>
2018-07-26 14:46 ` David Fetter <david@fetter.org>
2018-07-27 16:33 ` Andrew Gierth <andrew@tao11.riddles.org.uk>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox