agora inbox for [email protected]
help / color / mirror / Atom feedHuge memory consumption during vacuum (v.8.0)
67+ messages / 3 participants
[nested] [flat]
* Huge memory consumption during vacuum (v.8.0)
@ 2005-01-30 06:14 Oleg Bartunov <[email protected]>
2005-01-30 06:51 ` Re: Huge memory consumption during vacuum (v.8.0) Tom Lane <[email protected]>
0 siblings, 1 reply; 67+ messages in thread
From: Oleg Bartunov @ 2005-01-30 06:14 UTC (permalink / raw)
To: pgsql-hackers
Hi there,
seems I have a serious problem with vacuuming of rather big table
(500,000,000 rows) on dual Intel(R) Xeon(TM) CPU 2.40GHz, 1Gb RAM,
running Linux 2.6.7. I have PostgreSQL 8.0 release installed with
slightly changed postgresql.conf:
shared_buffers = 24576 # min 16, at least max_connections*2, 8KB each
maintenance_work_mem = 65536 # 16384 # min 1024, size in KB
checkpoint_segments = 12 #3 # in logfile segments, min 1, 16MB each
I tried run 'vacuumdb -v -z -f wsdb > vacuum-wsdb.log 2>&1&'
with default value of maintenance_work_mem but it was
too small for big table and I increased its value as Tom recommended.
But this change causes huge memory consumption - rather quickly memory
grew to 1Gb and after almost 42 hours of running (yes, it's still running)
postmaster eats more than 2Gb of RAM
20458 postgres 15 0 2462m 646m 204m D 37.5 63.9 744:38.74 postmaster
There are no messages in log file since start (just pg_* tables), so it's
difficult to say if there is some useful activity :)
The only non-standard action was installing 8.0 in neighbour with running
7.4.6 version. I run configure with different prefix and pgport specified
and use PGPORT, PGLIB, PGDATA, PATH modified to work with new postmaster.
I don't see any problem here.
Does anybody have experience vacuuming large database with 8.0 ?
table is very simple:
Table "public.usno"
Column | Type | Modifiers
--------+--------+-----------
ra | real |
dec | real |
bmag | real |
rmag | real |
ipix | bigint |
Indexes:
"ipix_ind" btree (ipix)
"radec_idx1" btree (ra, "dec")
Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: [email protected], http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Huge memory consumption during vacuum (v.8.0)
2005-01-30 06:14 Huge memory consumption during vacuum (v.8.0) Oleg Bartunov <[email protected]>
@ 2005-01-30 06:51 ` Tom Lane <[email protected]>
2005-01-30 06:58 ` Re: Huge memory consumption during vacuum (v.8.0) Oleg Bartunov <[email protected]>
0 siblings, 1 reply; 67+ messages in thread
From: Tom Lane @ 2005-01-30 06:51 UTC (permalink / raw)
To: Oleg Bartunov <[email protected]>; +Cc: pgsql-hackers
Oleg Bartunov <[email protected]> writes:
> I tried run 'vacuumdb -v -z -f wsdb > vacuum-wsdb.log 2>&1&'
I'm confused. The log trace you showed us before appeared to be from
a non-FULL vacuum, but here you're saying it's VACUUM FULL. Which is
it ... or did you change?
regards, tom lane
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Huge memory consumption during vacuum (v.8.0)
2005-01-30 06:14 Huge memory consumption during vacuum (v.8.0) Oleg Bartunov <[email protected]>
2005-01-30 06:51 ` Re: Huge memory consumption during vacuum (v.8.0) Tom Lane <[email protected]>
@ 2005-01-30 06:58 ` Oleg Bartunov <[email protected]>
2005-01-30 07:04 ` Re: Huge memory consumption during vacuum (v.8.0) Tom Lane <[email protected]>
0 siblings, 1 reply; 67+ messages in thread
From: Oleg Bartunov @ 2005-01-30 06:58 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: pgsql-hackers
On Sun, 30 Jan 2005, Tom Lane wrote:
> Oleg Bartunov <[email protected]> writes:
>> I tried run 'vacuumdb -v -z -f wsdb > vacuum-wsdb.log 2>&1&'
>
> I'm confused. The log trace you showed us before appeared to be from
> a non-FULL vacuum, but here you're saying it's VACUUM FULL. Which is
> it ... or did you change?
Yes, first time I tried vacuum from withing psql, next time I decided
to run vacuumdb and seems changed option.
>
> regards, tom lane
>
Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: [email protected], http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Huge memory consumption during vacuum (v.8.0)
2005-01-30 06:14 Huge memory consumption during vacuum (v.8.0) Oleg Bartunov <[email protected]>
2005-01-30 06:51 ` Re: Huge memory consumption during vacuum (v.8.0) Tom Lane <[email protected]>
2005-01-30 06:58 ` Re: Huge memory consumption during vacuum (v.8.0) Oleg Bartunov <[email protected]>
@ 2005-01-30 07:04 ` Tom Lane <[email protected]>
2005-01-30 07:10 ` Re: Huge memory consumption during vacuum (v.8.0) Oleg Bartunov <[email protected]>
0 siblings, 1 reply; 67+ messages in thread
From: Tom Lane @ 2005-01-30 07:04 UTC (permalink / raw)
To: Oleg Bartunov <[email protected]>; +Cc: pgsql-hackers
Oleg Bartunov <[email protected]> writes:
> On Sun, 30 Jan 2005, Tom Lane wrote:
>> I'm confused. The log trace you showed us before appeared to be from
>> a non-FULL vacuum, but here you're saying it's VACUUM FULL. Which is
>> it ... or did you change?
> Yes, first time I tried vacuum from withing psql, next time I decided
> to run vacuumdb and seems changed option.
Um. Well, a VACUUM FULL is going to build in-memory data structures
that represent *all* of the usable free space in a table. I don't
actually think that VACUUM FULL is useful on an enormous table ... you
want to keep after it with routine plain VACUUMs, instead.
Another possibility is to use CLUSTER or a rewriting ALTER TABLE to
shrink the space, but be aware that this requires a transient second
copy of the table and indexes.
regards, tom lane
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Huge memory consumption during vacuum (v.8.0)
2005-01-30 06:14 Huge memory consumption during vacuum (v.8.0) Oleg Bartunov <[email protected]>
2005-01-30 06:51 ` Re: Huge memory consumption during vacuum (v.8.0) Tom Lane <[email protected]>
2005-01-30 06:58 ` Re: Huge memory consumption during vacuum (v.8.0) Oleg Bartunov <[email protected]>
2005-01-30 07:04 ` Re: Huge memory consumption during vacuum (v.8.0) Tom Lane <[email protected]>
@ 2005-01-30 07:10 ` Oleg Bartunov <[email protected]>
2005-01-30 07:23 ` Re: Huge memory consumption during vacuum (v.8.0) Oleg Bartunov <[email protected]>
0 siblings, 1 reply; 67+ messages in thread
From: Oleg Bartunov @ 2005-01-30 07:10 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: pgsql-hackers
On Sun, 30 Jan 2005, Tom Lane wrote:
> Oleg Bartunov <[email protected]> writes:
>> On Sun, 30 Jan 2005, Tom Lane wrote:
>>> I'm confused. The log trace you showed us before appeared to be from
>>> a non-FULL vacuum, but here you're saying it's VACUUM FULL. Which is
>>> it ... or did you change?
>
>> Yes, first time I tried vacuum from withing psql, next time I decided
>> to run vacuumdb and seems changed option.
>
> Um. Well, a VACUUM FULL is going to build in-memory data structures
> that represent *all* of the usable free space in a table. I don't
> actually think that VACUUM FULL is useful on an enormous table ... you
> want to keep after it with routine plain VACUUMs, instead.
ok. I'll try without FULL, but if memory does not fail me postmaster was
also greedy. Let's see
>
> Another possibility is to use CLUSTER or a rewriting ALTER TABLE to
> shrink the space, but be aware that this requires a transient second
> copy of the table and indexes.
>
> regards, tom lane
>
Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: [email protected], http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Huge memory consumption during vacuum (v.8.0)
2005-01-30 06:14 Huge memory consumption during vacuum (v.8.0) Oleg Bartunov <[email protected]>
2005-01-30 06:51 ` Re: Huge memory consumption during vacuum (v.8.0) Tom Lane <[email protected]>
2005-01-30 06:58 ` Re: Huge memory consumption during vacuum (v.8.0) Oleg Bartunov <[email protected]>
2005-01-30 07:04 ` Re: Huge memory consumption during vacuum (v.8.0) Tom Lane <[email protected]>
2005-01-30 07:10 ` Re: Huge memory consumption during vacuum (v.8.0) Oleg Bartunov <[email protected]>
@ 2005-01-30 07:23 ` Oleg Bartunov <[email protected]>
2005-02-01 20:11 ` Re: Huge memory consumption during vacuum (v.8.0) Oleg Bartunov <[email protected]>
0 siblings, 1 reply; 67+ messages in thread
From: Oleg Bartunov @ 2005-01-30 07:23 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: pgsql-hackers
On Sun, 30 Jan 2005, Oleg Bartunov wrote:
> On Sun, 30 Jan 2005, Tom Lane wrote:
>
>> Oleg Bartunov <[email protected]> writes:
>>> On Sun, 30 Jan 2005, Tom Lane wrote:
>>>> I'm confused. The log trace you showed us before appeared to be from
>>>> a non-FULL vacuum, but here you're saying it's VACUUM FULL. Which is
>>>> it ... or did you change?
>>
>>> Yes, first time I tried vacuum from withing psql, next time I decided
>>> to run vacuumdb and seems changed option.
>>
>> Um. Well, a VACUUM FULL is going to build in-memory data structures
>> that represent *all* of the usable free space in a table. I don't
>> actually think that VACUUM FULL is useful on an enormous table ... you
>> want to keep after it with routine plain VACUUMs, instead.
>
> ok. I'll try without FULL, but if memory does not fail me postmaster was
> also greedy. Let's see
Seems, postmaster eats expected amount of memory now ! Will see how long
it will proceeded. Probably, my case should be documented somewhere.
>
>>
>> Another possibility is to use CLUSTER or a rewriting ALTER TABLE to
>> shrink the space, but be aware that this requires a transient second
>> copy of the table and indexes.
I aware, but I don't so much free space :)
Is there TODO for scaling VACUUM FULL ?
>>
>> regards, tom lane
>>
>
> Regards,
> Oleg
> _____________________________________________________________
> Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
> Sternberg Astronomical Institute, Moscow University (Russia)
> Internet: [email protected], http://www.sai.msu.su/~megera/
> phone: +007(095)939-16-83, +007(095)939-23-83
>
> ---------------------------(end of broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
>
> http://www.postgresql.org/docs/faq
>
Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: [email protected], http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Huge memory consumption during vacuum (v.8.0)
2005-01-30 06:14 Huge memory consumption during vacuum (v.8.0) Oleg Bartunov <[email protected]>
2005-01-30 06:51 ` Re: Huge memory consumption during vacuum (v.8.0) Tom Lane <[email protected]>
2005-01-30 06:58 ` Re: Huge memory consumption during vacuum (v.8.0) Oleg Bartunov <[email protected]>
2005-01-30 07:04 ` Re: Huge memory consumption during vacuum (v.8.0) Tom Lane <[email protected]>
2005-01-30 07:10 ` Re: Huge memory consumption during vacuum (v.8.0) Oleg Bartunov <[email protected]>
2005-01-30 07:23 ` Re: Huge memory consumption during vacuum (v.8.0) Oleg Bartunov <[email protected]>
@ 2005-02-01 20:11 ` Oleg Bartunov <[email protected]>
2005-02-01 21:26 ` Re: Huge memory consumption during vacuum (v.8.0) Tom Lane <[email protected]>
0 siblings, 1 reply; 67+ messages in thread
From: Oleg Bartunov @ 2005-02-01 20:11 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: pgsql-hackers
On Sun, 30 Jan 2005, Oleg Bartunov wrote:
>
> Seems, postmaster eats expected amount of memory now ! Will see how long
> it will proceeded. Probably, my case should be documented somewhere.
just to inform - vacuum took almost 48 hours !
>
>>
>>>
>>> Another possibility is to use CLUSTER or a rewriting ALTER TABLE to
>>> shrink the space, but be aware that this requires a transient second
>>> copy of the table and indexes.
>
> I aware, but I don't so much free space :)
I run
wsdb=# create table c_usno as select * from usno order by ipix;
to cluster table and notice that disk usage changing in discontinuous manner.
After some reduction I see no changes. postmaster is doing something
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
12723 postgres 18 0 207m 196m 204m D 21.6 19.4 52:17.12 postmaster
I see that pgsql_tmp/ contains files, looks like clustered table.
What postmaster is doing if disk usage doesn't changed ?
>
> Is there TODO for scaling VACUUM FULL ?
>
>
>
>>>
>>> regards, tom lane
>>>
>>
>> Regards,
>> Oleg
>> _____________________________________________________________
>> Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
>> Sternberg Astronomical Institute, Moscow University (Russia)
>> Internet: [email protected], http://www.sai.msu.su/~megera/
>> phone: +007(095)939-16-83, +007(095)939-23-83
>>
>> ---------------------------(end of broadcast)---------------------------
>> TIP 5: Have you checked our extensive FAQ?
>>
>> http://www.postgresql.org/docs/faq
>>
>
> Regards,
> Oleg
> _____________________________________________________________
> Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
> Sternberg Astronomical Institute, Moscow University (Russia)
> Internet: [email protected], http://www.sai.msu.su/~megera/
> phone: +007(095)939-16-83, +007(095)939-23-83
>
> ---------------------------(end of broadcast)---------------------------
> TIP 9: the planner will ignore your desire to choose an index scan if your
> joining column's datatypes do not match
>
Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: [email protected], http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Huge memory consumption during vacuum (v.8.0)
2005-01-30 06:14 Huge memory consumption during vacuum (v.8.0) Oleg Bartunov <[email protected]>
2005-01-30 06:51 ` Re: Huge memory consumption during vacuum (v.8.0) Tom Lane <[email protected]>
2005-01-30 06:58 ` Re: Huge memory consumption during vacuum (v.8.0) Oleg Bartunov <[email protected]>
2005-01-30 07:04 ` Re: Huge memory consumption during vacuum (v.8.0) Tom Lane <[email protected]>
2005-01-30 07:10 ` Re: Huge memory consumption during vacuum (v.8.0) Oleg Bartunov <[email protected]>
2005-01-30 07:23 ` Re: Huge memory consumption during vacuum (v.8.0) Oleg Bartunov <[email protected]>
2005-02-01 20:11 ` Re: Huge memory consumption during vacuum (v.8.0) Oleg Bartunov <[email protected]>
@ 2005-02-01 21:26 ` Tom Lane <[email protected]>
2005-02-01 21:30 ` Re: Huge memory consumption during vacuum (v.8.0) Oleg Bartunov <[email protected]>
2005-02-02 07:27 ` Re: Huge memory consumption during vacuum (v.8.0) Oleg Bartunov <[email protected]>
0 siblings, 2 replies; 67+ messages in thread
From: Tom Lane @ 2005-02-01 21:26 UTC (permalink / raw)
To: Oleg Bartunov <[email protected]>; +Cc: pgsql-hackers
Oleg Bartunov <[email protected]> writes:
> I see that pgsql_tmp/ contains files, looks like clustered table.
> What postmaster is doing if disk usage doesn't changed ?
Most likely doing a disk-based merge sort ...
regards, tom lane
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Huge memory consumption during vacuum (v.8.0)
2005-01-30 06:14 Huge memory consumption during vacuum (v.8.0) Oleg Bartunov <[email protected]>
2005-01-30 06:51 ` Re: Huge memory consumption during vacuum (v.8.0) Tom Lane <[email protected]>
2005-01-30 06:58 ` Re: Huge memory consumption during vacuum (v.8.0) Oleg Bartunov <[email protected]>
2005-01-30 07:04 ` Re: Huge memory consumption during vacuum (v.8.0) Tom Lane <[email protected]>
2005-01-30 07:10 ` Re: Huge memory consumption during vacuum (v.8.0) Oleg Bartunov <[email protected]>
2005-01-30 07:23 ` Re: Huge memory consumption during vacuum (v.8.0) Oleg Bartunov <[email protected]>
2005-02-01 20:11 ` Re: Huge memory consumption during vacuum (v.8.0) Oleg Bartunov <[email protected]>
2005-02-01 21:26 ` Re: Huge memory consumption during vacuum (v.8.0) Tom Lane <[email protected]>
@ 2005-02-01 21:30 ` Oleg Bartunov <[email protected]>
1 sibling, 0 replies; 67+ messages in thread
From: Oleg Bartunov @ 2005-02-01 21:30 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: pgsql-hackers
On Tue, 1 Feb 2005, Tom Lane wrote:
> Oleg Bartunov <[email protected]> writes:
>> I see that pgsql_tmp/ contains files, looks like clustered table.
>> What postmaster is doing if disk usage doesn't changed ?
>
> Most likely doing a disk-based merge sort ...
>
just interesting - multiway, in-place or just place merge sort ?
> regards, tom lane
>
Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: [email protected], http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83
^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Huge memory consumption during vacuum (v.8.0)
2005-01-30 06:14 Huge memory consumption during vacuum (v.8.0) Oleg Bartunov <[email protected]>
2005-01-30 06:51 ` Re: Huge memory consumption during vacuum (v.8.0) Tom Lane <[email protected]>
2005-01-30 06:58 ` Re: Huge memory consumption during vacuum (v.8.0) Oleg Bartunov <[email protected]>
2005-01-30 07:04 ` Re: Huge memory consumption during vacuum (v.8.0) Tom Lane <[email protected]>
2005-01-30 07:10 ` Re: Huge memory consumption during vacuum (v.8.0) Oleg Bartunov <[email protected]>
2005-01-30 07:23 ` Re: Huge memory consumption during vacuum (v.8.0) Oleg Bartunov <[email protected]>
2005-02-01 20:11 ` Re: Huge memory consumption during vacuum (v.8.0) Oleg Bartunov <[email protected]>
2005-02-01 21:26 ` Re: Huge memory consumption during vacuum (v.8.0) Tom Lane <[email protected]>
@ 2005-02-02 07:27 ` Oleg Bartunov <[email protected]>
1 sibling, 0 replies; 67+ messages in thread
From: Oleg Bartunov @ 2005-02-02 07:27 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: pgsql-hackers
On Tue, 1 Feb 2005, Tom Lane wrote:
> Oleg Bartunov <[email protected]> writes:
>> I see that pgsql_tmp/ contains files, looks like clustered table.
>> What postmaster is doing if disk usage doesn't changed ?
>
> Most likely doing a disk-based merge sort ...
>
so, it uses 'work_mem' as a buffer ?
> regards, tom lane
>
> ---------------------------(end of broadcast)---------------------------
> TIP 9: the planner will ignore your desire to choose an index scan if your
> joining column's datatypes do not match
>
Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: [email protected], http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 67 +++++++++++++++++++++++++++++++++++++--
src/include/nodes/execnodes.h | 2 ++
2 files changed, 67 insertions(+), 2 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..9a4063a 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,6 +199,8 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
+ int stopplan = node->as_whichplan;
+
for (;;)
{
PlanState *subnode;
@@ -207,7 +214,36 @@ ExecAppend(AppendState *node)
/*
* get a tuple from the subplan
*/
- result = ExecProcNode(subnode);
+ Assert(!node->as_finished[node->as_whichplan]);
+
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ /* Move to the next living node */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ /* No node is ready yet, return as not-ready */
+ if (node->as_whichplan == stopplan)
+ return;
+
+ /* Try the next node */
+ continue;
+ }
+
+ /* If not async, immediately wait for this subnode */
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ result = ExecConsumeResult((PlanState *) subnode);
if (!TupIsNull(result))
{
@@ -220,6 +256,31 @@ ExecAppend(AppendState *node)
return;
}
+ if (node->as_async)
+ {
+ node->as_finished[node->as_whichplan] = true;
+ stopplan = node->as_whichplan;
+
+ /* Find the next living subnode */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ if (node->as_whichplan != stopplan)
+ {
+ stopplan = node->as_whichplan;
+ continue;
+ }
+
+ /* All subnodes are exhausted. Finish this node. */
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
+ return;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
@@ -277,6 +338,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index b72decc..b0a86c5 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1177,6 +1177,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
2.9.2
----Next_Part(Mon_Aug_29_17_08_36_2016_213)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Mon_Aug_29_17_08_36_2016_213)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 67 +++++++++++++++++++++++++++++++++++++--
src/include/nodes/execnodes.h | 2 ++
2 files changed, 67 insertions(+), 2 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..9a4063a 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,6 +199,8 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
+ int stopplan = node->as_whichplan;
+
for (;;)
{
PlanState *subnode;
@@ -207,7 +214,36 @@ ExecAppend(AppendState *node)
/*
* get a tuple from the subplan
*/
- result = ExecProcNode(subnode);
+ Assert(!node->as_finished[node->as_whichplan]);
+
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ /* Move to the next living node */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ /* No node is ready yet, return as not-ready */
+ if (node->as_whichplan == stopplan)
+ return;
+
+ /* Try the next node */
+ continue;
+ }
+
+ /* If not async, immediately wait for this subnode */
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ result = ExecConsumeResult((PlanState *) subnode);
if (!TupIsNull(result))
{
@@ -220,6 +256,31 @@ ExecAppend(AppendState *node)
return;
}
+ if (node->as_async)
+ {
+ node->as_finished[node->as_whichplan] = true;
+ stopplan = node->as_whichplan;
+
+ /* Find the next living subnode */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ if (node->as_whichplan != stopplan)
+ {
+ stopplan = node->as_whichplan;
+ continue;
+ }
+
+ /* All subnodes are exhausted. Finish this node. */
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
+ return;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
@@ -277,6 +338,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index b72decc..b0a86c5 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1177,6 +1177,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
1.8.3.1
----Next_Part(Thu_Jul_21_18_50_07_2016_597)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Thu_Jul_21_18_50_07_2016_597)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 67 +++++++++++++++++++++++++++++++++++++--
src/include/nodes/execnodes.h | 2 ++
2 files changed, 67 insertions(+), 2 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..9a4063a 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,6 +199,8 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
+ int stopplan = node->as_whichplan;
+
for (;;)
{
PlanState *subnode;
@@ -207,7 +214,36 @@ ExecAppend(AppendState *node)
/*
* get a tuple from the subplan
*/
- result = ExecProcNode(subnode);
+ Assert(!node->as_finished[node->as_whichplan]);
+
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ /* Move to the next living node */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ /* No node is ready yet, return as not-ready */
+ if (node->as_whichplan == stopplan)
+ return;
+
+ /* Try the next node */
+ continue;
+ }
+
+ /* If not async, immediately wait for this subnode */
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ result = ExecConsumeResult((PlanState *) subnode);
if (!TupIsNull(result))
{
@@ -220,6 +256,31 @@ ExecAppend(AppendState *node)
return;
}
+ if (node->as_async)
+ {
+ node->as_finished[node->as_whichplan] = true;
+ stopplan = node->as_whichplan;
+
+ /* Find the next living subnode */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ if (node->as_whichplan != stopplan)
+ {
+ stopplan = node->as_whichplan;
+ continue;
+ }
+
+ /* All subnodes are exhausted. Finish this node. */
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
+ return;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
@@ -277,6 +338,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index b72decc..b0a86c5 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1177,6 +1177,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
1.8.3.1
----Next_Part(Thu_Jul_21_18_50_07_2016_597)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Thu_Jul_21_18_50_07_2016_597)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 94 ++++++++++++++++++++++++++++++++-------
src/include/nodes/execnodes.h | 2 +
2 files changed, 80 insertions(+), 16 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..1c0d26e 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,49 +199,104 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
- for (;;)
+ int n_notready = 0;
+ PlanState *subnode;
+ int i, n;
+
+ n = node->as_whichplan;
+
+ for (i = 0 ; i < node->as_nplans ; i++)
{
- PlanState *subnode;
TupleTableSlot *result;
+ if (node->as_async)
+ {
+ if (n >= node->as_nplans)
+ n = 0;
+
+ if (node->as_finished[n])
+ {
+ n++;
+ continue;
+ }
+ }
+
/*
* figure out which subplan we are currently processing
*/
- subnode = node->appendplans[node->as_whichplan];
+ subnode = node->appendplans[n];
/*
- * get a tuple from the subplan
+ * execute the subplan to get a result if it is not ready yet
*/
- result = ExecProcNode(subnode);
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ /*
+ * This subnode is not ready yet when asynchrony is not allowed,
+ * immediately wait for this subnode.
+ */
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ n_notready++;
+ n++;
+ continue;
+ }
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ Assert(subnode->result_ready);
+
+ result = ExecConsumeResult((PlanState *)subnode);
if (!TupIsNull(result))
{
/*
- * If the subplan gave us something then return it as-is. We do
- * NOT make use of the result slot that was set up in
- * ExecInitAppend; there's no need for it.
+ * If the subplan gave us something then return it
+ * as-is. We do NOT make use of the result slot that was
+ * set up in ExecInitAppend; there's no need for it.
*/
+ node->as_whichplan = n;
ExecReturnTuple(&node->ps, result);
return;
}
+ /* Tuple has been exhausted on this subnode */
+ if (node->as_async)
+ {
+ node->as_finished[n++] = true;
+ continue;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
* ExecInitAppend.
*/
if (ScanDirectionIsForward(node->ps.state->es_direction))
- node->as_whichplan++;
+ {
+ if (n >= node->as_nplans - 1)
+ break;
+ n++;
+ }
else
- node->as_whichplan--;
- if (!exec_append_initialize_next(node))
{
- ExecReturnTuple(&node->ps,
- ExecClearTuple(node->ps.ps_ResultTupleSlot));
- return;
+ if (n == 0)
+ break;
+ n--;
}
+ }
- /* Else loop back and try to get a tuple from the new subplan */
+ /*
+ * We are finished if reached here and no subnodes are not-ready
+ */
+ if (n_notready == 0)
+ {
+ node->as_whichplan = n;
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
}
}
@@ -277,6 +337,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 9121537..e4f2bb6 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1170,6 +1170,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
1.8.3.1
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: Text/Plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline; filename="gentbl.sql"
CREATE EXTENSION postgres_fdw;
CREATE TABLE pl (a int, b int);
CREATE TABLE cl1 (LIKE pl) INHERITS (pl);
CREATE TABLE cl2 (LIKE pl) INHERITS (pl);
CREATE TABLE cl3 (LIKE pl) INHERITS (pl);
CREATE TABLE cl4 (LIKE pl) INHERITS (pl);
INSERT INTO cl1 (SELECT a, a FROM generate_series(0000000, 0999999) a);
INSERT INTO cl2 (SELECT a, a FROM generate_series(1000000, 1999999) a);
INSERT INTO cl3 (SELECT a, a FROM generate_series(2000000, 2999999) a);
INSERT INTO cl4 (SELECT a, a FROM generate_series(3000000, 3999999) a);
CREATE TABLE t0 (LIKE pl);
INSERT INTO t0 (SELECT a, a FROM generate_series(0000000, 9999999) a);
CREATE SERVER sv0 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv1 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv2 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv3 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv4 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE USER MAPPING FOR public SERVER sv0;
CREATE USER MAPPING FOR public SERVER sv1;
CREATE USER MAPPING FOR public SERVER sv2;
CREATE USER MAPPING FOR public SERVER sv3;
CREATE USER MAPPING FOR public SERVER sv4;
CREATE FOREIGN TABLE ft10 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl1');
CREATE FOREIGN TABLE ft20 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl2');
CREATE FOREIGN TABLE ft30 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl3');
CREATE FOREIGN TABLE ft40 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl4');
CREATE FOREIGN TABLE ft11 (a int, b int) SERVER sv1 OPTIONS (table_name 'cl1');
CREATE FOREIGN TABLE ft22 (a int, b int) SERVER sv2 OPTIONS (table_name 'cl2');
CREATE FOREIGN TABLE ft33 (a int, b int) SERVER sv3 OPTIONS (table_name 'cl3');
CREATE FOREIGN TABLE ft44 (a int, b int) SERVER sv4 OPTIONS (table_name 'cl4');
CREATE TABLE pf0 (LIKE pl);
ALTER FOREIGN TABLE ft10 INHERIT pf0;
ALTER FOREIGN TABLE ft20 INHERIT pf0;
ALTER FOREIGN TABLE ft30 INHERIT pf0;
ALTER FOREIGN TABLE ft40 INHERIT pf0;
CREATE table pf1 (LIKE pl);
ALTER FOREIGN TABLE ft11 INHERIT pf1;
ALTER FOREIGN TABLE ft22 INHERIT pf1;
ALTER FOREIGN TABLE ft33 INHERIT pf1;
ALTER FOREIGN TABLE ft44 INHERIT pf1;
ANALYZE;
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: Text/Plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline; filename="testrun.sh"
#! /bin/bash
function do_test() {
echo $1
for i in $(seq 1 10); do psql postgres -c "set log_min_duration_statement = 0; set client_min_messages=log; select sum(a) from $1"; done | grep LOG
}
for i in "t0" "pl" "pf0" "pf1";
do
do_test $i
done
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Wed_Jul_06_16_29_14_2016_599)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 67 +++++++++++++++++++++++++++++++++++++--
src/include/nodes/execnodes.h | 2 ++
2 files changed, 67 insertions(+), 2 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..9a4063a 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,6 +199,8 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
+ int stopplan = node->as_whichplan;
+
for (;;)
{
PlanState *subnode;
@@ -207,7 +214,36 @@ ExecAppend(AppendState *node)
/*
* get a tuple from the subplan
*/
- result = ExecProcNode(subnode);
+ Assert(!node->as_finished[node->as_whichplan]);
+
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ /* Move to the next living node */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ /* No node is ready yet, return as not-ready */
+ if (node->as_whichplan == stopplan)
+ return;
+
+ /* Try the next node */
+ continue;
+ }
+
+ /* If not async, immediately wait for this subnode */
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ result = ExecConsumeResult((PlanState *) subnode);
if (!TupIsNull(result))
{
@@ -220,6 +256,31 @@ ExecAppend(AppendState *node)
return;
}
+ if (node->as_async)
+ {
+ node->as_finished[node->as_whichplan] = true;
+ stopplan = node->as_whichplan;
+
+ /* Find the next living subnode */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ if (node->as_whichplan != stopplan)
+ {
+ stopplan = node->as_whichplan;
+ continue;
+ }
+
+ /* All subnodes are exhausted. Finish this node. */
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
+ return;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
@@ -277,6 +338,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index b72decc..b0a86c5 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1177,6 +1177,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
1.8.3.1
----Next_Part(Thu_Jul_21_18_50_07_2016_597)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Thu_Jul_21_18_50_07_2016_597)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 94 ++++++++++++++++++++++++++++++++-------
src/include/nodes/execnodes.h | 2 +
2 files changed, 80 insertions(+), 16 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..1c0d26e 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,49 +199,104 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
- for (;;)
+ int n_notready = 0;
+ PlanState *subnode;
+ int i, n;
+
+ n = node->as_whichplan;
+
+ for (i = 0 ; i < node->as_nplans ; i++)
{
- PlanState *subnode;
TupleTableSlot *result;
+ if (node->as_async)
+ {
+ if (n >= node->as_nplans)
+ n = 0;
+
+ if (node->as_finished[n])
+ {
+ n++;
+ continue;
+ }
+ }
+
/*
* figure out which subplan we are currently processing
*/
- subnode = node->appendplans[node->as_whichplan];
+ subnode = node->appendplans[n];
/*
- * get a tuple from the subplan
+ * execute the subplan to get a result if it is not ready yet
*/
- result = ExecProcNode(subnode);
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ /*
+ * This subnode is not ready yet when asynchrony is not allowed,
+ * immediately wait for this subnode.
+ */
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ n_notready++;
+ n++;
+ continue;
+ }
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ Assert(subnode->result_ready);
+
+ result = ExecConsumeResult((PlanState *)subnode);
if (!TupIsNull(result))
{
/*
- * If the subplan gave us something then return it as-is. We do
- * NOT make use of the result slot that was set up in
- * ExecInitAppend; there's no need for it.
+ * If the subplan gave us something then return it
+ * as-is. We do NOT make use of the result slot that was
+ * set up in ExecInitAppend; there's no need for it.
*/
+ node->as_whichplan = n;
ExecReturnTuple(&node->ps, result);
return;
}
+ /* Tuple has been exhausted on this subnode */
+ if (node->as_async)
+ {
+ node->as_finished[n++] = true;
+ continue;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
* ExecInitAppend.
*/
if (ScanDirectionIsForward(node->ps.state->es_direction))
- node->as_whichplan++;
+ {
+ if (n >= node->as_nplans - 1)
+ break;
+ n++;
+ }
else
- node->as_whichplan--;
- if (!exec_append_initialize_next(node))
{
- ExecReturnTuple(&node->ps,
- ExecClearTuple(node->ps.ps_ResultTupleSlot));
- return;
+ if (n == 0)
+ break;
+ n--;
}
+ }
- /* Else loop back and try to get a tuple from the new subplan */
+ /*
+ * We are finished if reached here and no subnodes are not-ready
+ */
+ if (n_notready == 0)
+ {
+ node->as_whichplan = n;
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
}
}
@@ -277,6 +337,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 9121537..e4f2bb6 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1170,6 +1170,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
1.8.3.1
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: Text/Plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline; filename="gentbl.sql"
CREATE EXTENSION postgres_fdw;
CREATE TABLE pl (a int, b int);
CREATE TABLE cl1 (LIKE pl) INHERITS (pl);
CREATE TABLE cl2 (LIKE pl) INHERITS (pl);
CREATE TABLE cl3 (LIKE pl) INHERITS (pl);
CREATE TABLE cl4 (LIKE pl) INHERITS (pl);
INSERT INTO cl1 (SELECT a, a FROM generate_series(0000000, 0999999) a);
INSERT INTO cl2 (SELECT a, a FROM generate_series(1000000, 1999999) a);
INSERT INTO cl3 (SELECT a, a FROM generate_series(2000000, 2999999) a);
INSERT INTO cl4 (SELECT a, a FROM generate_series(3000000, 3999999) a);
CREATE TABLE t0 (LIKE pl);
INSERT INTO t0 (SELECT a, a FROM generate_series(0000000, 9999999) a);
CREATE SERVER sv0 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv1 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv2 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv3 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv4 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE USER MAPPING FOR public SERVER sv0;
CREATE USER MAPPING FOR public SERVER sv1;
CREATE USER MAPPING FOR public SERVER sv2;
CREATE USER MAPPING FOR public SERVER sv3;
CREATE USER MAPPING FOR public SERVER sv4;
CREATE FOREIGN TABLE ft10 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl1');
CREATE FOREIGN TABLE ft20 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl2');
CREATE FOREIGN TABLE ft30 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl3');
CREATE FOREIGN TABLE ft40 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl4');
CREATE FOREIGN TABLE ft11 (a int, b int) SERVER sv1 OPTIONS (table_name 'cl1');
CREATE FOREIGN TABLE ft22 (a int, b int) SERVER sv2 OPTIONS (table_name 'cl2');
CREATE FOREIGN TABLE ft33 (a int, b int) SERVER sv3 OPTIONS (table_name 'cl3');
CREATE FOREIGN TABLE ft44 (a int, b int) SERVER sv4 OPTIONS (table_name 'cl4');
CREATE TABLE pf0 (LIKE pl);
ALTER FOREIGN TABLE ft10 INHERIT pf0;
ALTER FOREIGN TABLE ft20 INHERIT pf0;
ALTER FOREIGN TABLE ft30 INHERIT pf0;
ALTER FOREIGN TABLE ft40 INHERIT pf0;
CREATE table pf1 (LIKE pl);
ALTER FOREIGN TABLE ft11 INHERIT pf1;
ALTER FOREIGN TABLE ft22 INHERIT pf1;
ALTER FOREIGN TABLE ft33 INHERIT pf1;
ALTER FOREIGN TABLE ft44 INHERIT pf1;
ANALYZE;
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: Text/Plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline; filename="testrun.sh"
#! /bin/bash
function do_test() {
echo $1
for i in $(seq 1 10); do psql postgres -c "set log_min_duration_statement = 0; set client_min_messages=log; select sum(a) from $1"; done | grep LOG
}
for i in "t0" "pl" "pf0" "pf1";
do
do_test $i
done
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Wed_Jul_06_16_29_14_2016_599)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 67 +++++++++++++++++++++++++++++++++++++--
src/include/nodes/execnodes.h | 2 ++
2 files changed, 67 insertions(+), 2 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..9a4063a 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,6 +199,8 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
+ int stopplan = node->as_whichplan;
+
for (;;)
{
PlanState *subnode;
@@ -207,7 +214,36 @@ ExecAppend(AppendState *node)
/*
* get a tuple from the subplan
*/
- result = ExecProcNode(subnode);
+ Assert(!node->as_finished[node->as_whichplan]);
+
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ /* Move to the next living node */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ /* No node is ready yet, return as not-ready */
+ if (node->as_whichplan == stopplan)
+ return;
+
+ /* Try the next node */
+ continue;
+ }
+
+ /* If not async, immediately wait for this subnode */
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ result = ExecConsumeResult((PlanState *) subnode);
if (!TupIsNull(result))
{
@@ -220,6 +256,31 @@ ExecAppend(AppendState *node)
return;
}
+ if (node->as_async)
+ {
+ node->as_finished[node->as_whichplan] = true;
+ stopplan = node->as_whichplan;
+
+ /* Find the next living subnode */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ if (node->as_whichplan != stopplan)
+ {
+ stopplan = node->as_whichplan;
+ continue;
+ }
+
+ /* All subnodes are exhausted. Finish this node. */
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
+ return;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
@@ -277,6 +338,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index b72decc..b0a86c5 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1177,6 +1177,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
2.9.2
----Next_Part(Mon_Aug_29_17_08_36_2016_213)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Mon_Aug_29_17_08_36_2016_213)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 94 ++++++++++++++++++++++++++++++++-------
src/include/nodes/execnodes.h | 2 +
2 files changed, 80 insertions(+), 16 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..1c0d26e 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,49 +199,104 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
- for (;;)
+ int n_notready = 0;
+ PlanState *subnode;
+ int i, n;
+
+ n = node->as_whichplan;
+
+ for (i = 0 ; i < node->as_nplans ; i++)
{
- PlanState *subnode;
TupleTableSlot *result;
+ if (node->as_async)
+ {
+ if (n >= node->as_nplans)
+ n = 0;
+
+ if (node->as_finished[n])
+ {
+ n++;
+ continue;
+ }
+ }
+
/*
* figure out which subplan we are currently processing
*/
- subnode = node->appendplans[node->as_whichplan];
+ subnode = node->appendplans[n];
/*
- * get a tuple from the subplan
+ * execute the subplan to get a result if it is not ready yet
*/
- result = ExecProcNode(subnode);
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ /*
+ * This subnode is not ready yet when asynchrony is not allowed,
+ * immediately wait for this subnode.
+ */
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ n_notready++;
+ n++;
+ continue;
+ }
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ Assert(subnode->result_ready);
+
+ result = ExecConsumeResult((PlanState *)subnode);
if (!TupIsNull(result))
{
/*
- * If the subplan gave us something then return it as-is. We do
- * NOT make use of the result slot that was set up in
- * ExecInitAppend; there's no need for it.
+ * If the subplan gave us something then return it
+ * as-is. We do NOT make use of the result slot that was
+ * set up in ExecInitAppend; there's no need for it.
*/
+ node->as_whichplan = n;
ExecReturnTuple(&node->ps, result);
return;
}
+ /* Tuple has been exhausted on this subnode */
+ if (node->as_async)
+ {
+ node->as_finished[n++] = true;
+ continue;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
* ExecInitAppend.
*/
if (ScanDirectionIsForward(node->ps.state->es_direction))
- node->as_whichplan++;
+ {
+ if (n >= node->as_nplans - 1)
+ break;
+ n++;
+ }
else
- node->as_whichplan--;
- if (!exec_append_initialize_next(node))
{
- ExecReturnTuple(&node->ps,
- ExecClearTuple(node->ps.ps_ResultTupleSlot));
- return;
+ if (n == 0)
+ break;
+ n--;
}
+ }
- /* Else loop back and try to get a tuple from the new subplan */
+ /*
+ * We are finished if reached here and no subnodes are not-ready
+ */
+ if (n_notready == 0)
+ {
+ node->as_whichplan = n;
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
}
}
@@ -277,6 +337,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 9121537..e4f2bb6 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1170,6 +1170,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
1.8.3.1
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: Text/Plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline; filename="gentbl.sql"
CREATE EXTENSION postgres_fdw;
CREATE TABLE pl (a int, b int);
CREATE TABLE cl1 (LIKE pl) INHERITS (pl);
CREATE TABLE cl2 (LIKE pl) INHERITS (pl);
CREATE TABLE cl3 (LIKE pl) INHERITS (pl);
CREATE TABLE cl4 (LIKE pl) INHERITS (pl);
INSERT INTO cl1 (SELECT a, a FROM generate_series(0000000, 0999999) a);
INSERT INTO cl2 (SELECT a, a FROM generate_series(1000000, 1999999) a);
INSERT INTO cl3 (SELECT a, a FROM generate_series(2000000, 2999999) a);
INSERT INTO cl4 (SELECT a, a FROM generate_series(3000000, 3999999) a);
CREATE TABLE t0 (LIKE pl);
INSERT INTO t0 (SELECT a, a FROM generate_series(0000000, 9999999) a);
CREATE SERVER sv0 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv1 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv2 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv3 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv4 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE USER MAPPING FOR public SERVER sv0;
CREATE USER MAPPING FOR public SERVER sv1;
CREATE USER MAPPING FOR public SERVER sv2;
CREATE USER MAPPING FOR public SERVER sv3;
CREATE USER MAPPING FOR public SERVER sv4;
CREATE FOREIGN TABLE ft10 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl1');
CREATE FOREIGN TABLE ft20 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl2');
CREATE FOREIGN TABLE ft30 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl3');
CREATE FOREIGN TABLE ft40 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl4');
CREATE FOREIGN TABLE ft11 (a int, b int) SERVER sv1 OPTIONS (table_name 'cl1');
CREATE FOREIGN TABLE ft22 (a int, b int) SERVER sv2 OPTIONS (table_name 'cl2');
CREATE FOREIGN TABLE ft33 (a int, b int) SERVER sv3 OPTIONS (table_name 'cl3');
CREATE FOREIGN TABLE ft44 (a int, b int) SERVER sv4 OPTIONS (table_name 'cl4');
CREATE TABLE pf0 (LIKE pl);
ALTER FOREIGN TABLE ft10 INHERIT pf0;
ALTER FOREIGN TABLE ft20 INHERIT pf0;
ALTER FOREIGN TABLE ft30 INHERIT pf0;
ALTER FOREIGN TABLE ft40 INHERIT pf0;
CREATE table pf1 (LIKE pl);
ALTER FOREIGN TABLE ft11 INHERIT pf1;
ALTER FOREIGN TABLE ft22 INHERIT pf1;
ALTER FOREIGN TABLE ft33 INHERIT pf1;
ALTER FOREIGN TABLE ft44 INHERIT pf1;
ANALYZE;
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: Text/Plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline; filename="testrun.sh"
#! /bin/bash
function do_test() {
echo $1
for i in $(seq 1 10); do psql postgres -c "set log_min_duration_statement = 0; set client_min_messages=log; select sum(a) from $1"; done | grep LOG
}
for i in "t0" "pl" "pf0" "pf1";
do
do_test $i
done
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Wed_Jul_06_16_29_14_2016_599)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 67 +++++++++++++++++++++++++++++++++++++--
src/include/nodes/execnodes.h | 2 ++
2 files changed, 67 insertions(+), 2 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..9a4063a 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,6 +199,8 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
+ int stopplan = node->as_whichplan;
+
for (;;)
{
PlanState *subnode;
@@ -207,7 +214,36 @@ ExecAppend(AppendState *node)
/*
* get a tuple from the subplan
*/
- result = ExecProcNode(subnode);
+ Assert(!node->as_finished[node->as_whichplan]);
+
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ /* Move to the next living node */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ /* No node is ready yet, return as not-ready */
+ if (node->as_whichplan == stopplan)
+ return;
+
+ /* Try the next node */
+ continue;
+ }
+
+ /* If not async, immediately wait for this subnode */
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ result = ExecConsumeResult((PlanState *) subnode);
if (!TupIsNull(result))
{
@@ -220,6 +256,31 @@ ExecAppend(AppendState *node)
return;
}
+ if (node->as_async)
+ {
+ node->as_finished[node->as_whichplan] = true;
+ stopplan = node->as_whichplan;
+
+ /* Find the next living subnode */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ if (node->as_whichplan != stopplan)
+ {
+ stopplan = node->as_whichplan;
+ continue;
+ }
+
+ /* All subnodes are exhausted. Finish this node. */
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
+ return;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
@@ -277,6 +338,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index b72decc..b0a86c5 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1177,6 +1177,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
1.8.3.1
----Next_Part(Thu_Jul_21_18_50_07_2016_597)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Thu_Jul_21_18_50_07_2016_597)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 67 +++++++++++++++++++++++++++++++++++++--
src/include/nodes/execnodes.h | 2 ++
2 files changed, 67 insertions(+), 2 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..9a4063a 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,6 +199,8 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
+ int stopplan = node->as_whichplan;
+
for (;;)
{
PlanState *subnode;
@@ -207,7 +214,36 @@ ExecAppend(AppendState *node)
/*
* get a tuple from the subplan
*/
- result = ExecProcNode(subnode);
+ Assert(!node->as_finished[node->as_whichplan]);
+
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ /* Move to the next living node */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ /* No node is ready yet, return as not-ready */
+ if (node->as_whichplan == stopplan)
+ return;
+
+ /* Try the next node */
+ continue;
+ }
+
+ /* If not async, immediately wait for this subnode */
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ result = ExecConsumeResult((PlanState *) subnode);
if (!TupIsNull(result))
{
@@ -220,6 +256,31 @@ ExecAppend(AppendState *node)
return;
}
+ if (node->as_async)
+ {
+ node->as_finished[node->as_whichplan] = true;
+ stopplan = node->as_whichplan;
+
+ /* Find the next living subnode */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ if (node->as_whichplan != stopplan)
+ {
+ stopplan = node->as_whichplan;
+ continue;
+ }
+
+ /* All subnodes are exhausted. Finish this node. */
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
+ return;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
@@ -277,6 +338,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index b72decc..b0a86c5 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1177,6 +1177,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
2.9.2
----Next_Part(Mon_Aug_29_17_08_36_2016_213)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Mon_Aug_29_17_08_36_2016_213)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 67 +++++++++++++++++++++++++++++++++++++--
src/include/nodes/execnodes.h | 2 ++
2 files changed, 67 insertions(+), 2 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..9a4063a 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,6 +199,8 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
+ int stopplan = node->as_whichplan;
+
for (;;)
{
PlanState *subnode;
@@ -207,7 +214,36 @@ ExecAppend(AppendState *node)
/*
* get a tuple from the subplan
*/
- result = ExecProcNode(subnode);
+ Assert(!node->as_finished[node->as_whichplan]);
+
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ /* Move to the next living node */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ /* No node is ready yet, return as not-ready */
+ if (node->as_whichplan == stopplan)
+ return;
+
+ /* Try the next node */
+ continue;
+ }
+
+ /* If not async, immediately wait for this subnode */
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ result = ExecConsumeResult((PlanState *) subnode);
if (!TupIsNull(result))
{
@@ -220,6 +256,31 @@ ExecAppend(AppendState *node)
return;
}
+ if (node->as_async)
+ {
+ node->as_finished[node->as_whichplan] = true;
+ stopplan = node->as_whichplan;
+
+ /* Find the next living subnode */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ if (node->as_whichplan != stopplan)
+ {
+ stopplan = node->as_whichplan;
+ continue;
+ }
+
+ /* All subnodes are exhausted. Finish this node. */
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
+ return;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
@@ -277,6 +338,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index b72decc..b0a86c5 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1177,6 +1177,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
2.9.2
----Next_Part(Mon_Aug_29_17_08_36_2016_213)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Mon_Aug_29_17_08_36_2016_213)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 67 +++++++++++++++++++++++++++++++++++++--
src/include/nodes/execnodes.h | 2 ++
2 files changed, 67 insertions(+), 2 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..9a4063a 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,6 +199,8 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
+ int stopplan = node->as_whichplan;
+
for (;;)
{
PlanState *subnode;
@@ -207,7 +214,36 @@ ExecAppend(AppendState *node)
/*
* get a tuple from the subplan
*/
- result = ExecProcNode(subnode);
+ Assert(!node->as_finished[node->as_whichplan]);
+
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ /* Move to the next living node */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ /* No node is ready yet, return as not-ready */
+ if (node->as_whichplan == stopplan)
+ return;
+
+ /* Try the next node */
+ continue;
+ }
+
+ /* If not async, immediately wait for this subnode */
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ result = ExecConsumeResult((PlanState *) subnode);
if (!TupIsNull(result))
{
@@ -220,6 +256,31 @@ ExecAppend(AppendState *node)
return;
}
+ if (node->as_async)
+ {
+ node->as_finished[node->as_whichplan] = true;
+ stopplan = node->as_whichplan;
+
+ /* Find the next living subnode */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ if (node->as_whichplan != stopplan)
+ {
+ stopplan = node->as_whichplan;
+ continue;
+ }
+
+ /* All subnodes are exhausted. Finish this node. */
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
+ return;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
@@ -277,6 +338,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index b72decc..b0a86c5 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1177,6 +1177,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
2.9.2
----Next_Part(Mon_Aug_29_17_08_36_2016_213)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Mon_Aug_29_17_08_36_2016_213)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 67 +++++++++++++++++++++++++++++++++++++--
src/include/nodes/execnodes.h | 2 ++
2 files changed, 67 insertions(+), 2 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..9a4063a 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,6 +199,8 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
+ int stopplan = node->as_whichplan;
+
for (;;)
{
PlanState *subnode;
@@ -207,7 +214,36 @@ ExecAppend(AppendState *node)
/*
* get a tuple from the subplan
*/
- result = ExecProcNode(subnode);
+ Assert(!node->as_finished[node->as_whichplan]);
+
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ /* Move to the next living node */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ /* No node is ready yet, return as not-ready */
+ if (node->as_whichplan == stopplan)
+ return;
+
+ /* Try the next node */
+ continue;
+ }
+
+ /* If not async, immediately wait for this subnode */
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ result = ExecConsumeResult((PlanState *) subnode);
if (!TupIsNull(result))
{
@@ -220,6 +256,31 @@ ExecAppend(AppendState *node)
return;
}
+ if (node->as_async)
+ {
+ node->as_finished[node->as_whichplan] = true;
+ stopplan = node->as_whichplan;
+
+ /* Find the next living subnode */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ if (node->as_whichplan != stopplan)
+ {
+ stopplan = node->as_whichplan;
+ continue;
+ }
+
+ /* All subnodes are exhausted. Finish this node. */
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
+ return;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
@@ -277,6 +338,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index b72decc..b0a86c5 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1177,6 +1177,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
1.8.3.1
----Next_Part(Thu_Jul_21_18_50_07_2016_597)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Thu_Jul_21_18_50_07_2016_597)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 94 ++++++++++++++++++++++++++++++++-------
src/include/nodes/execnodes.h | 2 +
2 files changed, 80 insertions(+), 16 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..1c0d26e 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,49 +199,104 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
- for (;;)
+ int n_notready = 0;
+ PlanState *subnode;
+ int i, n;
+
+ n = node->as_whichplan;
+
+ for (i = 0 ; i < node->as_nplans ; i++)
{
- PlanState *subnode;
TupleTableSlot *result;
+ if (node->as_async)
+ {
+ if (n >= node->as_nplans)
+ n = 0;
+
+ if (node->as_finished[n])
+ {
+ n++;
+ continue;
+ }
+ }
+
/*
* figure out which subplan we are currently processing
*/
- subnode = node->appendplans[node->as_whichplan];
+ subnode = node->appendplans[n];
/*
- * get a tuple from the subplan
+ * execute the subplan to get a result if it is not ready yet
*/
- result = ExecProcNode(subnode);
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ /*
+ * This subnode is not ready yet when asynchrony is not allowed,
+ * immediately wait for this subnode.
+ */
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ n_notready++;
+ n++;
+ continue;
+ }
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ Assert(subnode->result_ready);
+
+ result = ExecConsumeResult((PlanState *)subnode);
if (!TupIsNull(result))
{
/*
- * If the subplan gave us something then return it as-is. We do
- * NOT make use of the result slot that was set up in
- * ExecInitAppend; there's no need for it.
+ * If the subplan gave us something then return it
+ * as-is. We do NOT make use of the result slot that was
+ * set up in ExecInitAppend; there's no need for it.
*/
+ node->as_whichplan = n;
ExecReturnTuple(&node->ps, result);
return;
}
+ /* Tuple has been exhausted on this subnode */
+ if (node->as_async)
+ {
+ node->as_finished[n++] = true;
+ continue;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
* ExecInitAppend.
*/
if (ScanDirectionIsForward(node->ps.state->es_direction))
- node->as_whichplan++;
+ {
+ if (n >= node->as_nplans - 1)
+ break;
+ n++;
+ }
else
- node->as_whichplan--;
- if (!exec_append_initialize_next(node))
{
- ExecReturnTuple(&node->ps,
- ExecClearTuple(node->ps.ps_ResultTupleSlot));
- return;
+ if (n == 0)
+ break;
+ n--;
}
+ }
- /* Else loop back and try to get a tuple from the new subplan */
+ /*
+ * We are finished if reached here and no subnodes are not-ready
+ */
+ if (n_notready == 0)
+ {
+ node->as_whichplan = n;
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
}
}
@@ -277,6 +337,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 9121537..e4f2bb6 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1170,6 +1170,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
1.8.3.1
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: Text/Plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline; filename="gentbl.sql"
CREATE EXTENSION postgres_fdw;
CREATE TABLE pl (a int, b int);
CREATE TABLE cl1 (LIKE pl) INHERITS (pl);
CREATE TABLE cl2 (LIKE pl) INHERITS (pl);
CREATE TABLE cl3 (LIKE pl) INHERITS (pl);
CREATE TABLE cl4 (LIKE pl) INHERITS (pl);
INSERT INTO cl1 (SELECT a, a FROM generate_series(0000000, 0999999) a);
INSERT INTO cl2 (SELECT a, a FROM generate_series(1000000, 1999999) a);
INSERT INTO cl3 (SELECT a, a FROM generate_series(2000000, 2999999) a);
INSERT INTO cl4 (SELECT a, a FROM generate_series(3000000, 3999999) a);
CREATE TABLE t0 (LIKE pl);
INSERT INTO t0 (SELECT a, a FROM generate_series(0000000, 9999999) a);
CREATE SERVER sv0 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv1 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv2 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv3 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv4 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE USER MAPPING FOR public SERVER sv0;
CREATE USER MAPPING FOR public SERVER sv1;
CREATE USER MAPPING FOR public SERVER sv2;
CREATE USER MAPPING FOR public SERVER sv3;
CREATE USER MAPPING FOR public SERVER sv4;
CREATE FOREIGN TABLE ft10 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl1');
CREATE FOREIGN TABLE ft20 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl2');
CREATE FOREIGN TABLE ft30 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl3');
CREATE FOREIGN TABLE ft40 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl4');
CREATE FOREIGN TABLE ft11 (a int, b int) SERVER sv1 OPTIONS (table_name 'cl1');
CREATE FOREIGN TABLE ft22 (a int, b int) SERVER sv2 OPTIONS (table_name 'cl2');
CREATE FOREIGN TABLE ft33 (a int, b int) SERVER sv3 OPTIONS (table_name 'cl3');
CREATE FOREIGN TABLE ft44 (a int, b int) SERVER sv4 OPTIONS (table_name 'cl4');
CREATE TABLE pf0 (LIKE pl);
ALTER FOREIGN TABLE ft10 INHERIT pf0;
ALTER FOREIGN TABLE ft20 INHERIT pf0;
ALTER FOREIGN TABLE ft30 INHERIT pf0;
ALTER FOREIGN TABLE ft40 INHERIT pf0;
CREATE table pf1 (LIKE pl);
ALTER FOREIGN TABLE ft11 INHERIT pf1;
ALTER FOREIGN TABLE ft22 INHERIT pf1;
ALTER FOREIGN TABLE ft33 INHERIT pf1;
ALTER FOREIGN TABLE ft44 INHERIT pf1;
ANALYZE;
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: Text/Plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline; filename="testrun.sh"
#! /bin/bash
function do_test() {
echo $1
for i in $(seq 1 10); do psql postgres -c "set log_min_duration_statement = 0; set client_min_messages=log; select sum(a) from $1"; done | grep LOG
}
for i in "t0" "pl" "pf0" "pf1";
do
do_test $i
done
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Wed_Jul_06_16_29_14_2016_599)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 94 ++++++++++++++++++++++++++++++++-------
src/include/nodes/execnodes.h | 2 +
2 files changed, 80 insertions(+), 16 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..1c0d26e 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,49 +199,104 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
- for (;;)
+ int n_notready = 0;
+ PlanState *subnode;
+ int i, n;
+
+ n = node->as_whichplan;
+
+ for (i = 0 ; i < node->as_nplans ; i++)
{
- PlanState *subnode;
TupleTableSlot *result;
+ if (node->as_async)
+ {
+ if (n >= node->as_nplans)
+ n = 0;
+
+ if (node->as_finished[n])
+ {
+ n++;
+ continue;
+ }
+ }
+
/*
* figure out which subplan we are currently processing
*/
- subnode = node->appendplans[node->as_whichplan];
+ subnode = node->appendplans[n];
/*
- * get a tuple from the subplan
+ * execute the subplan to get a result if it is not ready yet
*/
- result = ExecProcNode(subnode);
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ /*
+ * This subnode is not ready yet when asynchrony is not allowed,
+ * immediately wait for this subnode.
+ */
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ n_notready++;
+ n++;
+ continue;
+ }
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ Assert(subnode->result_ready);
+
+ result = ExecConsumeResult((PlanState *)subnode);
if (!TupIsNull(result))
{
/*
- * If the subplan gave us something then return it as-is. We do
- * NOT make use of the result slot that was set up in
- * ExecInitAppend; there's no need for it.
+ * If the subplan gave us something then return it
+ * as-is. We do NOT make use of the result slot that was
+ * set up in ExecInitAppend; there's no need for it.
*/
+ node->as_whichplan = n;
ExecReturnTuple(&node->ps, result);
return;
}
+ /* Tuple has been exhausted on this subnode */
+ if (node->as_async)
+ {
+ node->as_finished[n++] = true;
+ continue;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
* ExecInitAppend.
*/
if (ScanDirectionIsForward(node->ps.state->es_direction))
- node->as_whichplan++;
+ {
+ if (n >= node->as_nplans - 1)
+ break;
+ n++;
+ }
else
- node->as_whichplan--;
- if (!exec_append_initialize_next(node))
{
- ExecReturnTuple(&node->ps,
- ExecClearTuple(node->ps.ps_ResultTupleSlot));
- return;
+ if (n == 0)
+ break;
+ n--;
}
+ }
- /* Else loop back and try to get a tuple from the new subplan */
+ /*
+ * We are finished if reached here and no subnodes are not-ready
+ */
+ if (n_notready == 0)
+ {
+ node->as_whichplan = n;
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
}
}
@@ -277,6 +337,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 9121537..e4f2bb6 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1170,6 +1170,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
1.8.3.1
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: Text/Plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline; filename="gentbl.sql"
CREATE EXTENSION postgres_fdw;
CREATE TABLE pl (a int, b int);
CREATE TABLE cl1 (LIKE pl) INHERITS (pl);
CREATE TABLE cl2 (LIKE pl) INHERITS (pl);
CREATE TABLE cl3 (LIKE pl) INHERITS (pl);
CREATE TABLE cl4 (LIKE pl) INHERITS (pl);
INSERT INTO cl1 (SELECT a, a FROM generate_series(0000000, 0999999) a);
INSERT INTO cl2 (SELECT a, a FROM generate_series(1000000, 1999999) a);
INSERT INTO cl3 (SELECT a, a FROM generate_series(2000000, 2999999) a);
INSERT INTO cl4 (SELECT a, a FROM generate_series(3000000, 3999999) a);
CREATE TABLE t0 (LIKE pl);
INSERT INTO t0 (SELECT a, a FROM generate_series(0000000, 9999999) a);
CREATE SERVER sv0 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv1 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv2 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv3 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv4 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE USER MAPPING FOR public SERVER sv0;
CREATE USER MAPPING FOR public SERVER sv1;
CREATE USER MAPPING FOR public SERVER sv2;
CREATE USER MAPPING FOR public SERVER sv3;
CREATE USER MAPPING FOR public SERVER sv4;
CREATE FOREIGN TABLE ft10 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl1');
CREATE FOREIGN TABLE ft20 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl2');
CREATE FOREIGN TABLE ft30 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl3');
CREATE FOREIGN TABLE ft40 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl4');
CREATE FOREIGN TABLE ft11 (a int, b int) SERVER sv1 OPTIONS (table_name 'cl1');
CREATE FOREIGN TABLE ft22 (a int, b int) SERVER sv2 OPTIONS (table_name 'cl2');
CREATE FOREIGN TABLE ft33 (a int, b int) SERVER sv3 OPTIONS (table_name 'cl3');
CREATE FOREIGN TABLE ft44 (a int, b int) SERVER sv4 OPTIONS (table_name 'cl4');
CREATE TABLE pf0 (LIKE pl);
ALTER FOREIGN TABLE ft10 INHERIT pf0;
ALTER FOREIGN TABLE ft20 INHERIT pf0;
ALTER FOREIGN TABLE ft30 INHERIT pf0;
ALTER FOREIGN TABLE ft40 INHERIT pf0;
CREATE table pf1 (LIKE pl);
ALTER FOREIGN TABLE ft11 INHERIT pf1;
ALTER FOREIGN TABLE ft22 INHERIT pf1;
ALTER FOREIGN TABLE ft33 INHERIT pf1;
ALTER FOREIGN TABLE ft44 INHERIT pf1;
ANALYZE;
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: Text/Plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline; filename="testrun.sh"
#! /bin/bash
function do_test() {
echo $1
for i in $(seq 1 10); do psql postgres -c "set log_min_duration_statement = 0; set client_min_messages=log; select sum(a) from $1"; done | grep LOG
}
for i in "t0" "pl" "pf0" "pf1";
do
do_test $i
done
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Wed_Jul_06_16_29_14_2016_599)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 67 +++++++++++++++++++++++++++++++++++++--
src/include/nodes/execnodes.h | 2 ++
2 files changed, 67 insertions(+), 2 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..9a4063a 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,6 +199,8 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
+ int stopplan = node->as_whichplan;
+
for (;;)
{
PlanState *subnode;
@@ -207,7 +214,36 @@ ExecAppend(AppendState *node)
/*
* get a tuple from the subplan
*/
- result = ExecProcNode(subnode);
+ Assert(!node->as_finished[node->as_whichplan]);
+
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ /* Move to the next living node */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ /* No node is ready yet, return as not-ready */
+ if (node->as_whichplan == stopplan)
+ return;
+
+ /* Try the next node */
+ continue;
+ }
+
+ /* If not async, immediately wait for this subnode */
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ result = ExecConsumeResult((PlanState *) subnode);
if (!TupIsNull(result))
{
@@ -220,6 +256,31 @@ ExecAppend(AppendState *node)
return;
}
+ if (node->as_async)
+ {
+ node->as_finished[node->as_whichplan] = true;
+ stopplan = node->as_whichplan;
+
+ /* Find the next living subnode */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ if (node->as_whichplan != stopplan)
+ {
+ stopplan = node->as_whichplan;
+ continue;
+ }
+
+ /* All subnodes are exhausted. Finish this node. */
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
+ return;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
@@ -277,6 +338,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index b72decc..b0a86c5 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1177,6 +1177,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
1.8.3.1
----Next_Part(Thu_Jul_21_18_50_07_2016_597)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Thu_Jul_21_18_50_07_2016_597)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 67 +++++++++++++++++++++++++++++++++++++--
src/include/nodes/execnodes.h | 2 ++
2 files changed, 67 insertions(+), 2 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..9a4063a 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,6 +199,8 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
+ int stopplan = node->as_whichplan;
+
for (;;)
{
PlanState *subnode;
@@ -207,7 +214,36 @@ ExecAppend(AppendState *node)
/*
* get a tuple from the subplan
*/
- result = ExecProcNode(subnode);
+ Assert(!node->as_finished[node->as_whichplan]);
+
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ /* Move to the next living node */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ /* No node is ready yet, return as not-ready */
+ if (node->as_whichplan == stopplan)
+ return;
+
+ /* Try the next node */
+ continue;
+ }
+
+ /* If not async, immediately wait for this subnode */
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ result = ExecConsumeResult((PlanState *) subnode);
if (!TupIsNull(result))
{
@@ -220,6 +256,31 @@ ExecAppend(AppendState *node)
return;
}
+ if (node->as_async)
+ {
+ node->as_finished[node->as_whichplan] = true;
+ stopplan = node->as_whichplan;
+
+ /* Find the next living subnode */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ if (node->as_whichplan != stopplan)
+ {
+ stopplan = node->as_whichplan;
+ continue;
+ }
+
+ /* All subnodes are exhausted. Finish this node. */
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
+ return;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
@@ -277,6 +338,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index b72decc..b0a86c5 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1177,6 +1177,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
2.9.2
----Next_Part(Mon_Aug_29_17_08_36_2016_213)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Mon_Aug_29_17_08_36_2016_213)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 67 +++++++++++++++++++++++++++++++++++++--
src/include/nodes/execnodes.h | 2 ++
2 files changed, 67 insertions(+), 2 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..9a4063a 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,6 +199,8 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
+ int stopplan = node->as_whichplan;
+
for (;;)
{
PlanState *subnode;
@@ -207,7 +214,36 @@ ExecAppend(AppendState *node)
/*
* get a tuple from the subplan
*/
- result = ExecProcNode(subnode);
+ Assert(!node->as_finished[node->as_whichplan]);
+
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ /* Move to the next living node */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ /* No node is ready yet, return as not-ready */
+ if (node->as_whichplan == stopplan)
+ return;
+
+ /* Try the next node */
+ continue;
+ }
+
+ /* If not async, immediately wait for this subnode */
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ result = ExecConsumeResult((PlanState *) subnode);
if (!TupIsNull(result))
{
@@ -220,6 +256,31 @@ ExecAppend(AppendState *node)
return;
}
+ if (node->as_async)
+ {
+ node->as_finished[node->as_whichplan] = true;
+ stopplan = node->as_whichplan;
+
+ /* Find the next living subnode */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ if (node->as_whichplan != stopplan)
+ {
+ stopplan = node->as_whichplan;
+ continue;
+ }
+
+ /* All subnodes are exhausted. Finish this node. */
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
+ return;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
@@ -277,6 +338,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index b72decc..b0a86c5 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1177,6 +1177,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
2.9.2
----Next_Part(Mon_Aug_29_17_08_36_2016_213)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Mon_Aug_29_17_08_36_2016_213)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 67 +++++++++++++++++++++++++++++++++++++--
src/include/nodes/execnodes.h | 2 ++
2 files changed, 67 insertions(+), 2 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..9a4063a 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,6 +199,8 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
+ int stopplan = node->as_whichplan;
+
for (;;)
{
PlanState *subnode;
@@ -207,7 +214,36 @@ ExecAppend(AppendState *node)
/*
* get a tuple from the subplan
*/
- result = ExecProcNode(subnode);
+ Assert(!node->as_finished[node->as_whichplan]);
+
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ /* Move to the next living node */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ /* No node is ready yet, return as not-ready */
+ if (node->as_whichplan == stopplan)
+ return;
+
+ /* Try the next node */
+ continue;
+ }
+
+ /* If not async, immediately wait for this subnode */
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ result = ExecConsumeResult((PlanState *) subnode);
if (!TupIsNull(result))
{
@@ -220,6 +256,31 @@ ExecAppend(AppendState *node)
return;
}
+ if (node->as_async)
+ {
+ node->as_finished[node->as_whichplan] = true;
+ stopplan = node->as_whichplan;
+
+ /* Find the next living subnode */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ if (node->as_whichplan != stopplan)
+ {
+ stopplan = node->as_whichplan;
+ continue;
+ }
+
+ /* All subnodes are exhausted. Finish this node. */
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
+ return;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
@@ -277,6 +338,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index b72decc..b0a86c5 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1177,6 +1177,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
2.9.2
----Next_Part(Mon_Aug_29_17_08_36_2016_213)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Mon_Aug_29_17_08_36_2016_213)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 94 ++++++++++++++++++++++++++++++++-------
src/include/nodes/execnodes.h | 2 +
2 files changed, 80 insertions(+), 16 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..1c0d26e 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,49 +199,104 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
- for (;;)
+ int n_notready = 0;
+ PlanState *subnode;
+ int i, n;
+
+ n = node->as_whichplan;
+
+ for (i = 0 ; i < node->as_nplans ; i++)
{
- PlanState *subnode;
TupleTableSlot *result;
+ if (node->as_async)
+ {
+ if (n >= node->as_nplans)
+ n = 0;
+
+ if (node->as_finished[n])
+ {
+ n++;
+ continue;
+ }
+ }
+
/*
* figure out which subplan we are currently processing
*/
- subnode = node->appendplans[node->as_whichplan];
+ subnode = node->appendplans[n];
/*
- * get a tuple from the subplan
+ * execute the subplan to get a result if it is not ready yet
*/
- result = ExecProcNode(subnode);
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ /*
+ * This subnode is not ready yet when asynchrony is not allowed,
+ * immediately wait for this subnode.
+ */
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ n_notready++;
+ n++;
+ continue;
+ }
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ Assert(subnode->result_ready);
+
+ result = ExecConsumeResult((PlanState *)subnode);
if (!TupIsNull(result))
{
/*
- * If the subplan gave us something then return it as-is. We do
- * NOT make use of the result slot that was set up in
- * ExecInitAppend; there's no need for it.
+ * If the subplan gave us something then return it
+ * as-is. We do NOT make use of the result slot that was
+ * set up in ExecInitAppend; there's no need for it.
*/
+ node->as_whichplan = n;
ExecReturnTuple(&node->ps, result);
return;
}
+ /* Tuple has been exhausted on this subnode */
+ if (node->as_async)
+ {
+ node->as_finished[n++] = true;
+ continue;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
* ExecInitAppend.
*/
if (ScanDirectionIsForward(node->ps.state->es_direction))
- node->as_whichplan++;
+ {
+ if (n >= node->as_nplans - 1)
+ break;
+ n++;
+ }
else
- node->as_whichplan--;
- if (!exec_append_initialize_next(node))
{
- ExecReturnTuple(&node->ps,
- ExecClearTuple(node->ps.ps_ResultTupleSlot));
- return;
+ if (n == 0)
+ break;
+ n--;
}
+ }
- /* Else loop back and try to get a tuple from the new subplan */
+ /*
+ * We are finished if reached here and no subnodes are not-ready
+ */
+ if (n_notready == 0)
+ {
+ node->as_whichplan = n;
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
}
}
@@ -277,6 +337,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 9121537..e4f2bb6 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1170,6 +1170,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
1.8.3.1
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: Text/Plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline; filename="gentbl.sql"
CREATE EXTENSION postgres_fdw;
CREATE TABLE pl (a int, b int);
CREATE TABLE cl1 (LIKE pl) INHERITS (pl);
CREATE TABLE cl2 (LIKE pl) INHERITS (pl);
CREATE TABLE cl3 (LIKE pl) INHERITS (pl);
CREATE TABLE cl4 (LIKE pl) INHERITS (pl);
INSERT INTO cl1 (SELECT a, a FROM generate_series(0000000, 0999999) a);
INSERT INTO cl2 (SELECT a, a FROM generate_series(1000000, 1999999) a);
INSERT INTO cl3 (SELECT a, a FROM generate_series(2000000, 2999999) a);
INSERT INTO cl4 (SELECT a, a FROM generate_series(3000000, 3999999) a);
CREATE TABLE t0 (LIKE pl);
INSERT INTO t0 (SELECT a, a FROM generate_series(0000000, 9999999) a);
CREATE SERVER sv0 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv1 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv2 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv3 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv4 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE USER MAPPING FOR public SERVER sv0;
CREATE USER MAPPING FOR public SERVER sv1;
CREATE USER MAPPING FOR public SERVER sv2;
CREATE USER MAPPING FOR public SERVER sv3;
CREATE USER MAPPING FOR public SERVER sv4;
CREATE FOREIGN TABLE ft10 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl1');
CREATE FOREIGN TABLE ft20 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl2');
CREATE FOREIGN TABLE ft30 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl3');
CREATE FOREIGN TABLE ft40 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl4');
CREATE FOREIGN TABLE ft11 (a int, b int) SERVER sv1 OPTIONS (table_name 'cl1');
CREATE FOREIGN TABLE ft22 (a int, b int) SERVER sv2 OPTIONS (table_name 'cl2');
CREATE FOREIGN TABLE ft33 (a int, b int) SERVER sv3 OPTIONS (table_name 'cl3');
CREATE FOREIGN TABLE ft44 (a int, b int) SERVER sv4 OPTIONS (table_name 'cl4');
CREATE TABLE pf0 (LIKE pl);
ALTER FOREIGN TABLE ft10 INHERIT pf0;
ALTER FOREIGN TABLE ft20 INHERIT pf0;
ALTER FOREIGN TABLE ft30 INHERIT pf0;
ALTER FOREIGN TABLE ft40 INHERIT pf0;
CREATE table pf1 (LIKE pl);
ALTER FOREIGN TABLE ft11 INHERIT pf1;
ALTER FOREIGN TABLE ft22 INHERIT pf1;
ALTER FOREIGN TABLE ft33 INHERIT pf1;
ALTER FOREIGN TABLE ft44 INHERIT pf1;
ANALYZE;
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: Text/Plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline; filename="testrun.sh"
#! /bin/bash
function do_test() {
echo $1
for i in $(seq 1 10); do psql postgres -c "set log_min_duration_statement = 0; set client_min_messages=log; select sum(a) from $1"; done | grep LOG
}
for i in "t0" "pl" "pf0" "pf1";
do
do_test $i
done
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Wed_Jul_06_16_29_14_2016_599)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 67 +++++++++++++++++++++++++++++++++++++--
src/include/nodes/execnodes.h | 2 ++
2 files changed, 67 insertions(+), 2 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..9a4063a 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,6 +199,8 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
+ int stopplan = node->as_whichplan;
+
for (;;)
{
PlanState *subnode;
@@ -207,7 +214,36 @@ ExecAppend(AppendState *node)
/*
* get a tuple from the subplan
*/
- result = ExecProcNode(subnode);
+ Assert(!node->as_finished[node->as_whichplan]);
+
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ /* Move to the next living node */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ /* No node is ready yet, return as not-ready */
+ if (node->as_whichplan == stopplan)
+ return;
+
+ /* Try the next node */
+ continue;
+ }
+
+ /* If not async, immediately wait for this subnode */
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ result = ExecConsumeResult((PlanState *) subnode);
if (!TupIsNull(result))
{
@@ -220,6 +256,31 @@ ExecAppend(AppendState *node)
return;
}
+ if (node->as_async)
+ {
+ node->as_finished[node->as_whichplan] = true;
+ stopplan = node->as_whichplan;
+
+ /* Find the next living subnode */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ if (node->as_whichplan != stopplan)
+ {
+ stopplan = node->as_whichplan;
+ continue;
+ }
+
+ /* All subnodes are exhausted. Finish this node. */
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
+ return;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
@@ -277,6 +338,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index b72decc..b0a86c5 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1177,6 +1177,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
2.9.2
----Next_Part(Mon_Aug_29_17_08_36_2016_213)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Mon_Aug_29_17_08_36_2016_213)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 67 +++++++++++++++++++++++++++++++++++++--
src/include/nodes/execnodes.h | 2 ++
2 files changed, 67 insertions(+), 2 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..9a4063a 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,6 +199,8 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
+ int stopplan = node->as_whichplan;
+
for (;;)
{
PlanState *subnode;
@@ -207,7 +214,36 @@ ExecAppend(AppendState *node)
/*
* get a tuple from the subplan
*/
- result = ExecProcNode(subnode);
+ Assert(!node->as_finished[node->as_whichplan]);
+
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ /* Move to the next living node */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ /* No node is ready yet, return as not-ready */
+ if (node->as_whichplan == stopplan)
+ return;
+
+ /* Try the next node */
+ continue;
+ }
+
+ /* If not async, immediately wait for this subnode */
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ result = ExecConsumeResult((PlanState *) subnode);
if (!TupIsNull(result))
{
@@ -220,6 +256,31 @@ ExecAppend(AppendState *node)
return;
}
+ if (node->as_async)
+ {
+ node->as_finished[node->as_whichplan] = true;
+ stopplan = node->as_whichplan;
+
+ /* Find the next living subnode */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ if (node->as_whichplan != stopplan)
+ {
+ stopplan = node->as_whichplan;
+ continue;
+ }
+
+ /* All subnodes are exhausted. Finish this node. */
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
+ return;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
@@ -277,6 +338,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index b72decc..b0a86c5 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1177,6 +1177,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
1.8.3.1
----Next_Part(Thu_Jul_21_18_50_07_2016_597)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Thu_Jul_21_18_50_07_2016_597)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 94 ++++++++++++++++++++++++++++++++-------
src/include/nodes/execnodes.h | 2 +
2 files changed, 80 insertions(+), 16 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..1c0d26e 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,49 +199,104 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
- for (;;)
+ int n_notready = 0;
+ PlanState *subnode;
+ int i, n;
+
+ n = node->as_whichplan;
+
+ for (i = 0 ; i < node->as_nplans ; i++)
{
- PlanState *subnode;
TupleTableSlot *result;
+ if (node->as_async)
+ {
+ if (n >= node->as_nplans)
+ n = 0;
+
+ if (node->as_finished[n])
+ {
+ n++;
+ continue;
+ }
+ }
+
/*
* figure out which subplan we are currently processing
*/
- subnode = node->appendplans[node->as_whichplan];
+ subnode = node->appendplans[n];
/*
- * get a tuple from the subplan
+ * execute the subplan to get a result if it is not ready yet
*/
- result = ExecProcNode(subnode);
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ /*
+ * This subnode is not ready yet when asynchrony is not allowed,
+ * immediately wait for this subnode.
+ */
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ n_notready++;
+ n++;
+ continue;
+ }
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ Assert(subnode->result_ready);
+
+ result = ExecConsumeResult((PlanState *)subnode);
if (!TupIsNull(result))
{
/*
- * If the subplan gave us something then return it as-is. We do
- * NOT make use of the result slot that was set up in
- * ExecInitAppend; there's no need for it.
+ * If the subplan gave us something then return it
+ * as-is. We do NOT make use of the result slot that was
+ * set up in ExecInitAppend; there's no need for it.
*/
+ node->as_whichplan = n;
ExecReturnTuple(&node->ps, result);
return;
}
+ /* Tuple has been exhausted on this subnode */
+ if (node->as_async)
+ {
+ node->as_finished[n++] = true;
+ continue;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
* ExecInitAppend.
*/
if (ScanDirectionIsForward(node->ps.state->es_direction))
- node->as_whichplan++;
+ {
+ if (n >= node->as_nplans - 1)
+ break;
+ n++;
+ }
else
- node->as_whichplan--;
- if (!exec_append_initialize_next(node))
{
- ExecReturnTuple(&node->ps,
- ExecClearTuple(node->ps.ps_ResultTupleSlot));
- return;
+ if (n == 0)
+ break;
+ n--;
}
+ }
- /* Else loop back and try to get a tuple from the new subplan */
+ /*
+ * We are finished if reached here and no subnodes are not-ready
+ */
+ if (n_notready == 0)
+ {
+ node->as_whichplan = n;
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
}
}
@@ -277,6 +337,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 9121537..e4f2bb6 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1170,6 +1170,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
1.8.3.1
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: Text/Plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline; filename="gentbl.sql"
CREATE EXTENSION postgres_fdw;
CREATE TABLE pl (a int, b int);
CREATE TABLE cl1 (LIKE pl) INHERITS (pl);
CREATE TABLE cl2 (LIKE pl) INHERITS (pl);
CREATE TABLE cl3 (LIKE pl) INHERITS (pl);
CREATE TABLE cl4 (LIKE pl) INHERITS (pl);
INSERT INTO cl1 (SELECT a, a FROM generate_series(0000000, 0999999) a);
INSERT INTO cl2 (SELECT a, a FROM generate_series(1000000, 1999999) a);
INSERT INTO cl3 (SELECT a, a FROM generate_series(2000000, 2999999) a);
INSERT INTO cl4 (SELECT a, a FROM generate_series(3000000, 3999999) a);
CREATE TABLE t0 (LIKE pl);
INSERT INTO t0 (SELECT a, a FROM generate_series(0000000, 9999999) a);
CREATE SERVER sv0 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv1 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv2 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv3 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv4 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE USER MAPPING FOR public SERVER sv0;
CREATE USER MAPPING FOR public SERVER sv1;
CREATE USER MAPPING FOR public SERVER sv2;
CREATE USER MAPPING FOR public SERVER sv3;
CREATE USER MAPPING FOR public SERVER sv4;
CREATE FOREIGN TABLE ft10 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl1');
CREATE FOREIGN TABLE ft20 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl2');
CREATE FOREIGN TABLE ft30 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl3');
CREATE FOREIGN TABLE ft40 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl4');
CREATE FOREIGN TABLE ft11 (a int, b int) SERVER sv1 OPTIONS (table_name 'cl1');
CREATE FOREIGN TABLE ft22 (a int, b int) SERVER sv2 OPTIONS (table_name 'cl2');
CREATE FOREIGN TABLE ft33 (a int, b int) SERVER sv3 OPTIONS (table_name 'cl3');
CREATE FOREIGN TABLE ft44 (a int, b int) SERVER sv4 OPTIONS (table_name 'cl4');
CREATE TABLE pf0 (LIKE pl);
ALTER FOREIGN TABLE ft10 INHERIT pf0;
ALTER FOREIGN TABLE ft20 INHERIT pf0;
ALTER FOREIGN TABLE ft30 INHERIT pf0;
ALTER FOREIGN TABLE ft40 INHERIT pf0;
CREATE table pf1 (LIKE pl);
ALTER FOREIGN TABLE ft11 INHERIT pf1;
ALTER FOREIGN TABLE ft22 INHERIT pf1;
ALTER FOREIGN TABLE ft33 INHERIT pf1;
ALTER FOREIGN TABLE ft44 INHERIT pf1;
ANALYZE;
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: Text/Plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline; filename="testrun.sh"
#! /bin/bash
function do_test() {
echo $1
for i in $(seq 1 10); do psql postgres -c "set log_min_duration_statement = 0; set client_min_messages=log; select sum(a) from $1"; done | grep LOG
}
for i in "t0" "pl" "pf0" "pf1";
do
do_test $i
done
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Wed_Jul_06_16_29_14_2016_599)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 67 +++++++++++++++++++++++++++++++++++++--
src/include/nodes/execnodes.h | 2 ++
2 files changed, 67 insertions(+), 2 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..9a4063a 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,6 +199,8 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
+ int stopplan = node->as_whichplan;
+
for (;;)
{
PlanState *subnode;
@@ -207,7 +214,36 @@ ExecAppend(AppendState *node)
/*
* get a tuple from the subplan
*/
- result = ExecProcNode(subnode);
+ Assert(!node->as_finished[node->as_whichplan]);
+
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ /* Move to the next living node */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ /* No node is ready yet, return as not-ready */
+ if (node->as_whichplan == stopplan)
+ return;
+
+ /* Try the next node */
+ continue;
+ }
+
+ /* If not async, immediately wait for this subnode */
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ result = ExecConsumeResult((PlanState *) subnode);
if (!TupIsNull(result))
{
@@ -220,6 +256,31 @@ ExecAppend(AppendState *node)
return;
}
+ if (node->as_async)
+ {
+ node->as_finished[node->as_whichplan] = true;
+ stopplan = node->as_whichplan;
+
+ /* Find the next living subnode */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ if (node->as_whichplan != stopplan)
+ {
+ stopplan = node->as_whichplan;
+ continue;
+ }
+
+ /* All subnodes are exhausted. Finish this node. */
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
+ return;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
@@ -277,6 +338,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index b72decc..b0a86c5 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1177,6 +1177,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
1.8.3.1
----Next_Part(Thu_Jul_21_18_50_07_2016_597)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Thu_Jul_21_18_50_07_2016_597)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 67 +++++++++++++++++++++++++++++++++++++--
src/include/nodes/execnodes.h | 2 ++
2 files changed, 67 insertions(+), 2 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..9a4063a 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,6 +199,8 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
+ int stopplan = node->as_whichplan;
+
for (;;)
{
PlanState *subnode;
@@ -207,7 +214,36 @@ ExecAppend(AppendState *node)
/*
* get a tuple from the subplan
*/
- result = ExecProcNode(subnode);
+ Assert(!node->as_finished[node->as_whichplan]);
+
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ /* Move to the next living node */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ /* No node is ready yet, return as not-ready */
+ if (node->as_whichplan == stopplan)
+ return;
+
+ /* Try the next node */
+ continue;
+ }
+
+ /* If not async, immediately wait for this subnode */
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ result = ExecConsumeResult((PlanState *) subnode);
if (!TupIsNull(result))
{
@@ -220,6 +256,31 @@ ExecAppend(AppendState *node)
return;
}
+ if (node->as_async)
+ {
+ node->as_finished[node->as_whichplan] = true;
+ stopplan = node->as_whichplan;
+
+ /* Find the next living subnode */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ if (node->as_whichplan != stopplan)
+ {
+ stopplan = node->as_whichplan;
+ continue;
+ }
+
+ /* All subnodes are exhausted. Finish this node. */
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
+ return;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
@@ -277,6 +338,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index b72decc..b0a86c5 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1177,6 +1177,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
2.9.2
----Next_Part(Mon_Aug_29_17_08_36_2016_213)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Mon_Aug_29_17_08_36_2016_213)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 67 +++++++++++++++++++++++++++++++++++++--
src/include/nodes/execnodes.h | 2 ++
2 files changed, 67 insertions(+), 2 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..9a4063a 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,6 +199,8 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
+ int stopplan = node->as_whichplan;
+
for (;;)
{
PlanState *subnode;
@@ -207,7 +214,36 @@ ExecAppend(AppendState *node)
/*
* get a tuple from the subplan
*/
- result = ExecProcNode(subnode);
+ Assert(!node->as_finished[node->as_whichplan]);
+
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ /* Move to the next living node */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ /* No node is ready yet, return as not-ready */
+ if (node->as_whichplan == stopplan)
+ return;
+
+ /* Try the next node */
+ continue;
+ }
+
+ /* If not async, immediately wait for this subnode */
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ result = ExecConsumeResult((PlanState *) subnode);
if (!TupIsNull(result))
{
@@ -220,6 +256,31 @@ ExecAppend(AppendState *node)
return;
}
+ if (node->as_async)
+ {
+ node->as_finished[node->as_whichplan] = true;
+ stopplan = node->as_whichplan;
+
+ /* Find the next living subnode */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ if (node->as_whichplan != stopplan)
+ {
+ stopplan = node->as_whichplan;
+ continue;
+ }
+
+ /* All subnodes are exhausted. Finish this node. */
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
+ return;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
@@ -277,6 +338,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index b72decc..b0a86c5 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1177,6 +1177,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
1.8.3.1
----Next_Part(Thu_Jul_21_18_50_07_2016_597)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Thu_Jul_21_18_50_07_2016_597)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 94 ++++++++++++++++++++++++++++++++-------
src/include/nodes/execnodes.h | 2 +
2 files changed, 80 insertions(+), 16 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..1c0d26e 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,49 +199,104 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
- for (;;)
+ int n_notready = 0;
+ PlanState *subnode;
+ int i, n;
+
+ n = node->as_whichplan;
+
+ for (i = 0 ; i < node->as_nplans ; i++)
{
- PlanState *subnode;
TupleTableSlot *result;
+ if (node->as_async)
+ {
+ if (n >= node->as_nplans)
+ n = 0;
+
+ if (node->as_finished[n])
+ {
+ n++;
+ continue;
+ }
+ }
+
/*
* figure out which subplan we are currently processing
*/
- subnode = node->appendplans[node->as_whichplan];
+ subnode = node->appendplans[n];
/*
- * get a tuple from the subplan
+ * execute the subplan to get a result if it is not ready yet
*/
- result = ExecProcNode(subnode);
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ /*
+ * This subnode is not ready yet when asynchrony is not allowed,
+ * immediately wait for this subnode.
+ */
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ n_notready++;
+ n++;
+ continue;
+ }
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ Assert(subnode->result_ready);
+
+ result = ExecConsumeResult((PlanState *)subnode);
if (!TupIsNull(result))
{
/*
- * If the subplan gave us something then return it as-is. We do
- * NOT make use of the result slot that was set up in
- * ExecInitAppend; there's no need for it.
+ * If the subplan gave us something then return it
+ * as-is. We do NOT make use of the result slot that was
+ * set up in ExecInitAppend; there's no need for it.
*/
+ node->as_whichplan = n;
ExecReturnTuple(&node->ps, result);
return;
}
+ /* Tuple has been exhausted on this subnode */
+ if (node->as_async)
+ {
+ node->as_finished[n++] = true;
+ continue;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
* ExecInitAppend.
*/
if (ScanDirectionIsForward(node->ps.state->es_direction))
- node->as_whichplan++;
+ {
+ if (n >= node->as_nplans - 1)
+ break;
+ n++;
+ }
else
- node->as_whichplan--;
- if (!exec_append_initialize_next(node))
{
- ExecReturnTuple(&node->ps,
- ExecClearTuple(node->ps.ps_ResultTupleSlot));
- return;
+ if (n == 0)
+ break;
+ n--;
}
+ }
- /* Else loop back and try to get a tuple from the new subplan */
+ /*
+ * We are finished if reached here and no subnodes are not-ready
+ */
+ if (n_notready == 0)
+ {
+ node->as_whichplan = n;
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
}
}
@@ -277,6 +337,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 9121537..e4f2bb6 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1170,6 +1170,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
1.8.3.1
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: Text/Plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline; filename="gentbl.sql"
CREATE EXTENSION postgres_fdw;
CREATE TABLE pl (a int, b int);
CREATE TABLE cl1 (LIKE pl) INHERITS (pl);
CREATE TABLE cl2 (LIKE pl) INHERITS (pl);
CREATE TABLE cl3 (LIKE pl) INHERITS (pl);
CREATE TABLE cl4 (LIKE pl) INHERITS (pl);
INSERT INTO cl1 (SELECT a, a FROM generate_series(0000000, 0999999) a);
INSERT INTO cl2 (SELECT a, a FROM generate_series(1000000, 1999999) a);
INSERT INTO cl3 (SELECT a, a FROM generate_series(2000000, 2999999) a);
INSERT INTO cl4 (SELECT a, a FROM generate_series(3000000, 3999999) a);
CREATE TABLE t0 (LIKE pl);
INSERT INTO t0 (SELECT a, a FROM generate_series(0000000, 9999999) a);
CREATE SERVER sv0 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv1 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv2 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv3 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv4 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE USER MAPPING FOR public SERVER sv0;
CREATE USER MAPPING FOR public SERVER sv1;
CREATE USER MAPPING FOR public SERVER sv2;
CREATE USER MAPPING FOR public SERVER sv3;
CREATE USER MAPPING FOR public SERVER sv4;
CREATE FOREIGN TABLE ft10 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl1');
CREATE FOREIGN TABLE ft20 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl2');
CREATE FOREIGN TABLE ft30 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl3');
CREATE FOREIGN TABLE ft40 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl4');
CREATE FOREIGN TABLE ft11 (a int, b int) SERVER sv1 OPTIONS (table_name 'cl1');
CREATE FOREIGN TABLE ft22 (a int, b int) SERVER sv2 OPTIONS (table_name 'cl2');
CREATE FOREIGN TABLE ft33 (a int, b int) SERVER sv3 OPTIONS (table_name 'cl3');
CREATE FOREIGN TABLE ft44 (a int, b int) SERVER sv4 OPTIONS (table_name 'cl4');
CREATE TABLE pf0 (LIKE pl);
ALTER FOREIGN TABLE ft10 INHERIT pf0;
ALTER FOREIGN TABLE ft20 INHERIT pf0;
ALTER FOREIGN TABLE ft30 INHERIT pf0;
ALTER FOREIGN TABLE ft40 INHERIT pf0;
CREATE table pf1 (LIKE pl);
ALTER FOREIGN TABLE ft11 INHERIT pf1;
ALTER FOREIGN TABLE ft22 INHERIT pf1;
ALTER FOREIGN TABLE ft33 INHERIT pf1;
ALTER FOREIGN TABLE ft44 INHERIT pf1;
ANALYZE;
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: Text/Plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline; filename="testrun.sh"
#! /bin/bash
function do_test() {
echo $1
for i in $(seq 1 10); do psql postgres -c "set log_min_duration_statement = 0; set client_min_messages=log; select sum(a) from $1"; done | grep LOG
}
for i in "t0" "pl" "pf0" "pf1";
do
do_test $i
done
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Wed_Jul_06_16_29_14_2016_599)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 67 +++++++++++++++++++++++++++++++++++++--
src/include/nodes/execnodes.h | 2 ++
2 files changed, 67 insertions(+), 2 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..9a4063a 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,6 +199,8 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
+ int stopplan = node->as_whichplan;
+
for (;;)
{
PlanState *subnode;
@@ -207,7 +214,36 @@ ExecAppend(AppendState *node)
/*
* get a tuple from the subplan
*/
- result = ExecProcNode(subnode);
+ Assert(!node->as_finished[node->as_whichplan]);
+
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ /* Move to the next living node */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ /* No node is ready yet, return as not-ready */
+ if (node->as_whichplan == stopplan)
+ return;
+
+ /* Try the next node */
+ continue;
+ }
+
+ /* If not async, immediately wait for this subnode */
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ result = ExecConsumeResult((PlanState *) subnode);
if (!TupIsNull(result))
{
@@ -220,6 +256,31 @@ ExecAppend(AppendState *node)
return;
}
+ if (node->as_async)
+ {
+ node->as_finished[node->as_whichplan] = true;
+ stopplan = node->as_whichplan;
+
+ /* Find the next living subnode */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ if (node->as_whichplan != stopplan)
+ {
+ stopplan = node->as_whichplan;
+ continue;
+ }
+
+ /* All subnodes are exhausted. Finish this node. */
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
+ return;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
@@ -277,6 +338,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index b72decc..b0a86c5 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1177,6 +1177,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
2.9.2
----Next_Part(Mon_Aug_29_17_08_36_2016_213)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Mon_Aug_29_17_08_36_2016_213)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 67 +++++++++++++++++++++++++++++++++++++--
src/include/nodes/execnodes.h | 2 ++
2 files changed, 67 insertions(+), 2 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..9a4063a 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,6 +199,8 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
+ int stopplan = node->as_whichplan;
+
for (;;)
{
PlanState *subnode;
@@ -207,7 +214,36 @@ ExecAppend(AppendState *node)
/*
* get a tuple from the subplan
*/
- result = ExecProcNode(subnode);
+ Assert(!node->as_finished[node->as_whichplan]);
+
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ /* Move to the next living node */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ /* No node is ready yet, return as not-ready */
+ if (node->as_whichplan == stopplan)
+ return;
+
+ /* Try the next node */
+ continue;
+ }
+
+ /* If not async, immediately wait for this subnode */
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ result = ExecConsumeResult((PlanState *) subnode);
if (!TupIsNull(result))
{
@@ -220,6 +256,31 @@ ExecAppend(AppendState *node)
return;
}
+ if (node->as_async)
+ {
+ node->as_finished[node->as_whichplan] = true;
+ stopplan = node->as_whichplan;
+
+ /* Find the next living subnode */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ if (node->as_whichplan != stopplan)
+ {
+ stopplan = node->as_whichplan;
+ continue;
+ }
+
+ /* All subnodes are exhausted. Finish this node. */
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
+ return;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
@@ -277,6 +338,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index b72decc..b0a86c5 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1177,6 +1177,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
1.8.3.1
----Next_Part(Thu_Jul_21_18_50_07_2016_597)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Thu_Jul_21_18_50_07_2016_597)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 67 +++++++++++++++++++++++++++++++++++++--
src/include/nodes/execnodes.h | 2 ++
2 files changed, 67 insertions(+), 2 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..9a4063a 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,6 +199,8 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
+ int stopplan = node->as_whichplan;
+
for (;;)
{
PlanState *subnode;
@@ -207,7 +214,36 @@ ExecAppend(AppendState *node)
/*
* get a tuple from the subplan
*/
- result = ExecProcNode(subnode);
+ Assert(!node->as_finished[node->as_whichplan]);
+
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ /* Move to the next living node */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ /* No node is ready yet, return as not-ready */
+ if (node->as_whichplan == stopplan)
+ return;
+
+ /* Try the next node */
+ continue;
+ }
+
+ /* If not async, immediately wait for this subnode */
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ result = ExecConsumeResult((PlanState *) subnode);
if (!TupIsNull(result))
{
@@ -220,6 +256,31 @@ ExecAppend(AppendState *node)
return;
}
+ if (node->as_async)
+ {
+ node->as_finished[node->as_whichplan] = true;
+ stopplan = node->as_whichplan;
+
+ /* Find the next living subnode */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ if (node->as_whichplan != stopplan)
+ {
+ stopplan = node->as_whichplan;
+ continue;
+ }
+
+ /* All subnodes are exhausted. Finish this node. */
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
+ return;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
@@ -277,6 +338,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index b72decc..b0a86c5 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1177,6 +1177,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
2.9.2
----Next_Part(Mon_Aug_29_17_08_36_2016_213)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Mon_Aug_29_17_08_36_2016_213)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 67 +++++++++++++++++++++++++++++++++++++--
src/include/nodes/execnodes.h | 2 ++
2 files changed, 67 insertions(+), 2 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..9a4063a 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,6 +199,8 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
+ int stopplan = node->as_whichplan;
+
for (;;)
{
PlanState *subnode;
@@ -207,7 +214,36 @@ ExecAppend(AppendState *node)
/*
* get a tuple from the subplan
*/
- result = ExecProcNode(subnode);
+ Assert(!node->as_finished[node->as_whichplan]);
+
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ /* Move to the next living node */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ /* No node is ready yet, return as not-ready */
+ if (node->as_whichplan == stopplan)
+ return;
+
+ /* Try the next node */
+ continue;
+ }
+
+ /* If not async, immediately wait for this subnode */
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ result = ExecConsumeResult((PlanState *) subnode);
if (!TupIsNull(result))
{
@@ -220,6 +256,31 @@ ExecAppend(AppendState *node)
return;
}
+ if (node->as_async)
+ {
+ node->as_finished[node->as_whichplan] = true;
+ stopplan = node->as_whichplan;
+
+ /* Find the next living subnode */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ if (node->as_whichplan != stopplan)
+ {
+ stopplan = node->as_whichplan;
+ continue;
+ }
+
+ /* All subnodes are exhausted. Finish this node. */
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
+ return;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
@@ -277,6 +338,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index b72decc..b0a86c5 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1177,6 +1177,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
1.8.3.1
----Next_Part(Thu_Jul_21_18_50_07_2016_597)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Thu_Jul_21_18_50_07_2016_597)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 94 ++++++++++++++++++++++++++++++++-------
src/include/nodes/execnodes.h | 2 +
2 files changed, 80 insertions(+), 16 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..1c0d26e 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,49 +199,104 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
- for (;;)
+ int n_notready = 0;
+ PlanState *subnode;
+ int i, n;
+
+ n = node->as_whichplan;
+
+ for (i = 0 ; i < node->as_nplans ; i++)
{
- PlanState *subnode;
TupleTableSlot *result;
+ if (node->as_async)
+ {
+ if (n >= node->as_nplans)
+ n = 0;
+
+ if (node->as_finished[n])
+ {
+ n++;
+ continue;
+ }
+ }
+
/*
* figure out which subplan we are currently processing
*/
- subnode = node->appendplans[node->as_whichplan];
+ subnode = node->appendplans[n];
/*
- * get a tuple from the subplan
+ * execute the subplan to get a result if it is not ready yet
*/
- result = ExecProcNode(subnode);
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ /*
+ * This subnode is not ready yet when asynchrony is not allowed,
+ * immediately wait for this subnode.
+ */
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ n_notready++;
+ n++;
+ continue;
+ }
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ Assert(subnode->result_ready);
+
+ result = ExecConsumeResult((PlanState *)subnode);
if (!TupIsNull(result))
{
/*
- * If the subplan gave us something then return it as-is. We do
- * NOT make use of the result slot that was set up in
- * ExecInitAppend; there's no need for it.
+ * If the subplan gave us something then return it
+ * as-is. We do NOT make use of the result slot that was
+ * set up in ExecInitAppend; there's no need for it.
*/
+ node->as_whichplan = n;
ExecReturnTuple(&node->ps, result);
return;
}
+ /* Tuple has been exhausted on this subnode */
+ if (node->as_async)
+ {
+ node->as_finished[n++] = true;
+ continue;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
* ExecInitAppend.
*/
if (ScanDirectionIsForward(node->ps.state->es_direction))
- node->as_whichplan++;
+ {
+ if (n >= node->as_nplans - 1)
+ break;
+ n++;
+ }
else
- node->as_whichplan--;
- if (!exec_append_initialize_next(node))
{
- ExecReturnTuple(&node->ps,
- ExecClearTuple(node->ps.ps_ResultTupleSlot));
- return;
+ if (n == 0)
+ break;
+ n--;
}
+ }
- /* Else loop back and try to get a tuple from the new subplan */
+ /*
+ * We are finished if reached here and no subnodes are not-ready
+ */
+ if (n_notready == 0)
+ {
+ node->as_whichplan = n;
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
}
}
@@ -277,6 +337,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 9121537..e4f2bb6 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1170,6 +1170,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
1.8.3.1
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: Text/Plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline; filename="gentbl.sql"
CREATE EXTENSION postgres_fdw;
CREATE TABLE pl (a int, b int);
CREATE TABLE cl1 (LIKE pl) INHERITS (pl);
CREATE TABLE cl2 (LIKE pl) INHERITS (pl);
CREATE TABLE cl3 (LIKE pl) INHERITS (pl);
CREATE TABLE cl4 (LIKE pl) INHERITS (pl);
INSERT INTO cl1 (SELECT a, a FROM generate_series(0000000, 0999999) a);
INSERT INTO cl2 (SELECT a, a FROM generate_series(1000000, 1999999) a);
INSERT INTO cl3 (SELECT a, a FROM generate_series(2000000, 2999999) a);
INSERT INTO cl4 (SELECT a, a FROM generate_series(3000000, 3999999) a);
CREATE TABLE t0 (LIKE pl);
INSERT INTO t0 (SELECT a, a FROM generate_series(0000000, 9999999) a);
CREATE SERVER sv0 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv1 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv2 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv3 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv4 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE USER MAPPING FOR public SERVER sv0;
CREATE USER MAPPING FOR public SERVER sv1;
CREATE USER MAPPING FOR public SERVER sv2;
CREATE USER MAPPING FOR public SERVER sv3;
CREATE USER MAPPING FOR public SERVER sv4;
CREATE FOREIGN TABLE ft10 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl1');
CREATE FOREIGN TABLE ft20 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl2');
CREATE FOREIGN TABLE ft30 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl3');
CREATE FOREIGN TABLE ft40 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl4');
CREATE FOREIGN TABLE ft11 (a int, b int) SERVER sv1 OPTIONS (table_name 'cl1');
CREATE FOREIGN TABLE ft22 (a int, b int) SERVER sv2 OPTIONS (table_name 'cl2');
CREATE FOREIGN TABLE ft33 (a int, b int) SERVER sv3 OPTIONS (table_name 'cl3');
CREATE FOREIGN TABLE ft44 (a int, b int) SERVER sv4 OPTIONS (table_name 'cl4');
CREATE TABLE pf0 (LIKE pl);
ALTER FOREIGN TABLE ft10 INHERIT pf0;
ALTER FOREIGN TABLE ft20 INHERIT pf0;
ALTER FOREIGN TABLE ft30 INHERIT pf0;
ALTER FOREIGN TABLE ft40 INHERIT pf0;
CREATE table pf1 (LIKE pl);
ALTER FOREIGN TABLE ft11 INHERIT pf1;
ALTER FOREIGN TABLE ft22 INHERIT pf1;
ALTER FOREIGN TABLE ft33 INHERIT pf1;
ALTER FOREIGN TABLE ft44 INHERIT pf1;
ANALYZE;
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: Text/Plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline; filename="testrun.sh"
#! /bin/bash
function do_test() {
echo $1
for i in $(seq 1 10); do psql postgres -c "set log_min_duration_statement = 0; set client_min_messages=log; select sum(a) from $1"; done | grep LOG
}
for i in "t0" "pl" "pf0" "pf1";
do
do_test $i
done
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Wed_Jul_06_16_29_14_2016_599)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 94 ++++++++++++++++++++++++++++++++-------
src/include/nodes/execnodes.h | 2 +
2 files changed, 80 insertions(+), 16 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..1c0d26e 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,49 +199,104 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
- for (;;)
+ int n_notready = 0;
+ PlanState *subnode;
+ int i, n;
+
+ n = node->as_whichplan;
+
+ for (i = 0 ; i < node->as_nplans ; i++)
{
- PlanState *subnode;
TupleTableSlot *result;
+ if (node->as_async)
+ {
+ if (n >= node->as_nplans)
+ n = 0;
+
+ if (node->as_finished[n])
+ {
+ n++;
+ continue;
+ }
+ }
+
/*
* figure out which subplan we are currently processing
*/
- subnode = node->appendplans[node->as_whichplan];
+ subnode = node->appendplans[n];
/*
- * get a tuple from the subplan
+ * execute the subplan to get a result if it is not ready yet
*/
- result = ExecProcNode(subnode);
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ /*
+ * This subnode is not ready yet when asynchrony is not allowed,
+ * immediately wait for this subnode.
+ */
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ n_notready++;
+ n++;
+ continue;
+ }
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ Assert(subnode->result_ready);
+
+ result = ExecConsumeResult((PlanState *)subnode);
if (!TupIsNull(result))
{
/*
- * If the subplan gave us something then return it as-is. We do
- * NOT make use of the result slot that was set up in
- * ExecInitAppend; there's no need for it.
+ * If the subplan gave us something then return it
+ * as-is. We do NOT make use of the result slot that was
+ * set up in ExecInitAppend; there's no need for it.
*/
+ node->as_whichplan = n;
ExecReturnTuple(&node->ps, result);
return;
}
+ /* Tuple has been exhausted on this subnode */
+ if (node->as_async)
+ {
+ node->as_finished[n++] = true;
+ continue;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
* ExecInitAppend.
*/
if (ScanDirectionIsForward(node->ps.state->es_direction))
- node->as_whichplan++;
+ {
+ if (n >= node->as_nplans - 1)
+ break;
+ n++;
+ }
else
- node->as_whichplan--;
- if (!exec_append_initialize_next(node))
{
- ExecReturnTuple(&node->ps,
- ExecClearTuple(node->ps.ps_ResultTupleSlot));
- return;
+ if (n == 0)
+ break;
+ n--;
}
+ }
- /* Else loop back and try to get a tuple from the new subplan */
+ /*
+ * We are finished if reached here and no subnodes are not-ready
+ */
+ if (n_notready == 0)
+ {
+ node->as_whichplan = n;
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
}
}
@@ -277,6 +337,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 9121537..e4f2bb6 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1170,6 +1170,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
1.8.3.1
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: Text/Plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline; filename="gentbl.sql"
CREATE EXTENSION postgres_fdw;
CREATE TABLE pl (a int, b int);
CREATE TABLE cl1 (LIKE pl) INHERITS (pl);
CREATE TABLE cl2 (LIKE pl) INHERITS (pl);
CREATE TABLE cl3 (LIKE pl) INHERITS (pl);
CREATE TABLE cl4 (LIKE pl) INHERITS (pl);
INSERT INTO cl1 (SELECT a, a FROM generate_series(0000000, 0999999) a);
INSERT INTO cl2 (SELECT a, a FROM generate_series(1000000, 1999999) a);
INSERT INTO cl3 (SELECT a, a FROM generate_series(2000000, 2999999) a);
INSERT INTO cl4 (SELECT a, a FROM generate_series(3000000, 3999999) a);
CREATE TABLE t0 (LIKE pl);
INSERT INTO t0 (SELECT a, a FROM generate_series(0000000, 9999999) a);
CREATE SERVER sv0 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv1 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv2 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv3 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv4 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE USER MAPPING FOR public SERVER sv0;
CREATE USER MAPPING FOR public SERVER sv1;
CREATE USER MAPPING FOR public SERVER sv2;
CREATE USER MAPPING FOR public SERVER sv3;
CREATE USER MAPPING FOR public SERVER sv4;
CREATE FOREIGN TABLE ft10 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl1');
CREATE FOREIGN TABLE ft20 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl2');
CREATE FOREIGN TABLE ft30 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl3');
CREATE FOREIGN TABLE ft40 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl4');
CREATE FOREIGN TABLE ft11 (a int, b int) SERVER sv1 OPTIONS (table_name 'cl1');
CREATE FOREIGN TABLE ft22 (a int, b int) SERVER sv2 OPTIONS (table_name 'cl2');
CREATE FOREIGN TABLE ft33 (a int, b int) SERVER sv3 OPTIONS (table_name 'cl3');
CREATE FOREIGN TABLE ft44 (a int, b int) SERVER sv4 OPTIONS (table_name 'cl4');
CREATE TABLE pf0 (LIKE pl);
ALTER FOREIGN TABLE ft10 INHERIT pf0;
ALTER FOREIGN TABLE ft20 INHERIT pf0;
ALTER FOREIGN TABLE ft30 INHERIT pf0;
ALTER FOREIGN TABLE ft40 INHERIT pf0;
CREATE table pf1 (LIKE pl);
ALTER FOREIGN TABLE ft11 INHERIT pf1;
ALTER FOREIGN TABLE ft22 INHERIT pf1;
ALTER FOREIGN TABLE ft33 INHERIT pf1;
ALTER FOREIGN TABLE ft44 INHERIT pf1;
ANALYZE;
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: Text/Plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline; filename="testrun.sh"
#! /bin/bash
function do_test() {
echo $1
for i in $(seq 1 10); do psql postgres -c "set log_min_duration_statement = 0; set client_min_messages=log; select sum(a) from $1"; done | grep LOG
}
for i in "t0" "pl" "pf0" "pf1";
do
do_test $i
done
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Wed_Jul_06_16_29_14_2016_599)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 94 ++++++++++++++++++++++++++++++++-------
src/include/nodes/execnodes.h | 2 +
2 files changed, 80 insertions(+), 16 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..1c0d26e 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,49 +199,104 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
- for (;;)
+ int n_notready = 0;
+ PlanState *subnode;
+ int i, n;
+
+ n = node->as_whichplan;
+
+ for (i = 0 ; i < node->as_nplans ; i++)
{
- PlanState *subnode;
TupleTableSlot *result;
+ if (node->as_async)
+ {
+ if (n >= node->as_nplans)
+ n = 0;
+
+ if (node->as_finished[n])
+ {
+ n++;
+ continue;
+ }
+ }
+
/*
* figure out which subplan we are currently processing
*/
- subnode = node->appendplans[node->as_whichplan];
+ subnode = node->appendplans[n];
/*
- * get a tuple from the subplan
+ * execute the subplan to get a result if it is not ready yet
*/
- result = ExecProcNode(subnode);
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ /*
+ * This subnode is not ready yet when asynchrony is not allowed,
+ * immediately wait for this subnode.
+ */
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ n_notready++;
+ n++;
+ continue;
+ }
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ Assert(subnode->result_ready);
+
+ result = ExecConsumeResult((PlanState *)subnode);
if (!TupIsNull(result))
{
/*
- * If the subplan gave us something then return it as-is. We do
- * NOT make use of the result slot that was set up in
- * ExecInitAppend; there's no need for it.
+ * If the subplan gave us something then return it
+ * as-is. We do NOT make use of the result slot that was
+ * set up in ExecInitAppend; there's no need for it.
*/
+ node->as_whichplan = n;
ExecReturnTuple(&node->ps, result);
return;
}
+ /* Tuple has been exhausted on this subnode */
+ if (node->as_async)
+ {
+ node->as_finished[n++] = true;
+ continue;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
* ExecInitAppend.
*/
if (ScanDirectionIsForward(node->ps.state->es_direction))
- node->as_whichplan++;
+ {
+ if (n >= node->as_nplans - 1)
+ break;
+ n++;
+ }
else
- node->as_whichplan--;
- if (!exec_append_initialize_next(node))
{
- ExecReturnTuple(&node->ps,
- ExecClearTuple(node->ps.ps_ResultTupleSlot));
- return;
+ if (n == 0)
+ break;
+ n--;
}
+ }
- /* Else loop back and try to get a tuple from the new subplan */
+ /*
+ * We are finished if reached here and no subnodes are not-ready
+ */
+ if (n_notready == 0)
+ {
+ node->as_whichplan = n;
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
}
}
@@ -277,6 +337,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 9121537..e4f2bb6 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1170,6 +1170,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
1.8.3.1
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: Text/Plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline; filename="gentbl.sql"
CREATE EXTENSION postgres_fdw;
CREATE TABLE pl (a int, b int);
CREATE TABLE cl1 (LIKE pl) INHERITS (pl);
CREATE TABLE cl2 (LIKE pl) INHERITS (pl);
CREATE TABLE cl3 (LIKE pl) INHERITS (pl);
CREATE TABLE cl4 (LIKE pl) INHERITS (pl);
INSERT INTO cl1 (SELECT a, a FROM generate_series(0000000, 0999999) a);
INSERT INTO cl2 (SELECT a, a FROM generate_series(1000000, 1999999) a);
INSERT INTO cl3 (SELECT a, a FROM generate_series(2000000, 2999999) a);
INSERT INTO cl4 (SELECT a, a FROM generate_series(3000000, 3999999) a);
CREATE TABLE t0 (LIKE pl);
INSERT INTO t0 (SELECT a, a FROM generate_series(0000000, 9999999) a);
CREATE SERVER sv0 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv1 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv2 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv3 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv4 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE USER MAPPING FOR public SERVER sv0;
CREATE USER MAPPING FOR public SERVER sv1;
CREATE USER MAPPING FOR public SERVER sv2;
CREATE USER MAPPING FOR public SERVER sv3;
CREATE USER MAPPING FOR public SERVER sv4;
CREATE FOREIGN TABLE ft10 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl1');
CREATE FOREIGN TABLE ft20 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl2');
CREATE FOREIGN TABLE ft30 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl3');
CREATE FOREIGN TABLE ft40 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl4');
CREATE FOREIGN TABLE ft11 (a int, b int) SERVER sv1 OPTIONS (table_name 'cl1');
CREATE FOREIGN TABLE ft22 (a int, b int) SERVER sv2 OPTIONS (table_name 'cl2');
CREATE FOREIGN TABLE ft33 (a int, b int) SERVER sv3 OPTIONS (table_name 'cl3');
CREATE FOREIGN TABLE ft44 (a int, b int) SERVER sv4 OPTIONS (table_name 'cl4');
CREATE TABLE pf0 (LIKE pl);
ALTER FOREIGN TABLE ft10 INHERIT pf0;
ALTER FOREIGN TABLE ft20 INHERIT pf0;
ALTER FOREIGN TABLE ft30 INHERIT pf0;
ALTER FOREIGN TABLE ft40 INHERIT pf0;
CREATE table pf1 (LIKE pl);
ALTER FOREIGN TABLE ft11 INHERIT pf1;
ALTER FOREIGN TABLE ft22 INHERIT pf1;
ALTER FOREIGN TABLE ft33 INHERIT pf1;
ALTER FOREIGN TABLE ft44 INHERIT pf1;
ANALYZE;
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: Text/Plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline; filename="testrun.sh"
#! /bin/bash
function do_test() {
echo $1
for i in $(seq 1 10); do psql postgres -c "set log_min_duration_statement = 0; set client_min_messages=log; select sum(a) from $1"; done | grep LOG
}
for i in "t0" "pl" "pf0" "pf1";
do
do_test $i
done
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Wed_Jul_06_16_29_14_2016_599)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 94 ++++++++++++++++++++++++++++++++-------
src/include/nodes/execnodes.h | 2 +
2 files changed, 80 insertions(+), 16 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..1c0d26e 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,49 +199,104 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
- for (;;)
+ int n_notready = 0;
+ PlanState *subnode;
+ int i, n;
+
+ n = node->as_whichplan;
+
+ for (i = 0 ; i < node->as_nplans ; i++)
{
- PlanState *subnode;
TupleTableSlot *result;
+ if (node->as_async)
+ {
+ if (n >= node->as_nplans)
+ n = 0;
+
+ if (node->as_finished[n])
+ {
+ n++;
+ continue;
+ }
+ }
+
/*
* figure out which subplan we are currently processing
*/
- subnode = node->appendplans[node->as_whichplan];
+ subnode = node->appendplans[n];
/*
- * get a tuple from the subplan
+ * execute the subplan to get a result if it is not ready yet
*/
- result = ExecProcNode(subnode);
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ /*
+ * This subnode is not ready yet when asynchrony is not allowed,
+ * immediately wait for this subnode.
+ */
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ n_notready++;
+ n++;
+ continue;
+ }
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ Assert(subnode->result_ready);
+
+ result = ExecConsumeResult((PlanState *)subnode);
if (!TupIsNull(result))
{
/*
- * If the subplan gave us something then return it as-is. We do
- * NOT make use of the result slot that was set up in
- * ExecInitAppend; there's no need for it.
+ * If the subplan gave us something then return it
+ * as-is. We do NOT make use of the result slot that was
+ * set up in ExecInitAppend; there's no need for it.
*/
+ node->as_whichplan = n;
ExecReturnTuple(&node->ps, result);
return;
}
+ /* Tuple has been exhausted on this subnode */
+ if (node->as_async)
+ {
+ node->as_finished[n++] = true;
+ continue;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
* ExecInitAppend.
*/
if (ScanDirectionIsForward(node->ps.state->es_direction))
- node->as_whichplan++;
+ {
+ if (n >= node->as_nplans - 1)
+ break;
+ n++;
+ }
else
- node->as_whichplan--;
- if (!exec_append_initialize_next(node))
{
- ExecReturnTuple(&node->ps,
- ExecClearTuple(node->ps.ps_ResultTupleSlot));
- return;
+ if (n == 0)
+ break;
+ n--;
}
+ }
- /* Else loop back and try to get a tuple from the new subplan */
+ /*
+ * We are finished if reached here and no subnodes are not-ready
+ */
+ if (n_notready == 0)
+ {
+ node->as_whichplan = n;
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
}
}
@@ -277,6 +337,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 9121537..e4f2bb6 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1170,6 +1170,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
1.8.3.1
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: Text/Plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline; filename="gentbl.sql"
CREATE EXTENSION postgres_fdw;
CREATE TABLE pl (a int, b int);
CREATE TABLE cl1 (LIKE pl) INHERITS (pl);
CREATE TABLE cl2 (LIKE pl) INHERITS (pl);
CREATE TABLE cl3 (LIKE pl) INHERITS (pl);
CREATE TABLE cl4 (LIKE pl) INHERITS (pl);
INSERT INTO cl1 (SELECT a, a FROM generate_series(0000000, 0999999) a);
INSERT INTO cl2 (SELECT a, a FROM generate_series(1000000, 1999999) a);
INSERT INTO cl3 (SELECT a, a FROM generate_series(2000000, 2999999) a);
INSERT INTO cl4 (SELECT a, a FROM generate_series(3000000, 3999999) a);
CREATE TABLE t0 (LIKE pl);
INSERT INTO t0 (SELECT a, a FROM generate_series(0000000, 9999999) a);
CREATE SERVER sv0 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv1 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv2 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv3 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv4 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE USER MAPPING FOR public SERVER sv0;
CREATE USER MAPPING FOR public SERVER sv1;
CREATE USER MAPPING FOR public SERVER sv2;
CREATE USER MAPPING FOR public SERVER sv3;
CREATE USER MAPPING FOR public SERVER sv4;
CREATE FOREIGN TABLE ft10 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl1');
CREATE FOREIGN TABLE ft20 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl2');
CREATE FOREIGN TABLE ft30 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl3');
CREATE FOREIGN TABLE ft40 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl4');
CREATE FOREIGN TABLE ft11 (a int, b int) SERVER sv1 OPTIONS (table_name 'cl1');
CREATE FOREIGN TABLE ft22 (a int, b int) SERVER sv2 OPTIONS (table_name 'cl2');
CREATE FOREIGN TABLE ft33 (a int, b int) SERVER sv3 OPTIONS (table_name 'cl3');
CREATE FOREIGN TABLE ft44 (a int, b int) SERVER sv4 OPTIONS (table_name 'cl4');
CREATE TABLE pf0 (LIKE pl);
ALTER FOREIGN TABLE ft10 INHERIT pf0;
ALTER FOREIGN TABLE ft20 INHERIT pf0;
ALTER FOREIGN TABLE ft30 INHERIT pf0;
ALTER FOREIGN TABLE ft40 INHERIT pf0;
CREATE table pf1 (LIKE pl);
ALTER FOREIGN TABLE ft11 INHERIT pf1;
ALTER FOREIGN TABLE ft22 INHERIT pf1;
ALTER FOREIGN TABLE ft33 INHERIT pf1;
ALTER FOREIGN TABLE ft44 INHERIT pf1;
ANALYZE;
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: Text/Plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline; filename="testrun.sh"
#! /bin/bash
function do_test() {
echo $1
for i in $(seq 1 10); do psql postgres -c "set log_min_duration_statement = 0; set client_min_messages=log; select sum(a) from $1"; done | grep LOG
}
for i in "t0" "pl" "pf0" "pf1";
do
do_test $i
done
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Wed_Jul_06_16_29_14_2016_599)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 67 +++++++++++++++++++++++++++++++++++++--
src/include/nodes/execnodes.h | 2 ++
2 files changed, 67 insertions(+), 2 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..9a4063a 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,6 +199,8 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
+ int stopplan = node->as_whichplan;
+
for (;;)
{
PlanState *subnode;
@@ -207,7 +214,36 @@ ExecAppend(AppendState *node)
/*
* get a tuple from the subplan
*/
- result = ExecProcNode(subnode);
+ Assert(!node->as_finished[node->as_whichplan]);
+
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ /* Move to the next living node */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ /* No node is ready yet, return as not-ready */
+ if (node->as_whichplan == stopplan)
+ return;
+
+ /* Try the next node */
+ continue;
+ }
+
+ /* If not async, immediately wait for this subnode */
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ result = ExecConsumeResult((PlanState *) subnode);
if (!TupIsNull(result))
{
@@ -220,6 +256,31 @@ ExecAppend(AppendState *node)
return;
}
+ if (node->as_async)
+ {
+ node->as_finished[node->as_whichplan] = true;
+ stopplan = node->as_whichplan;
+
+ /* Find the next living subnode */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ if (node->as_whichplan != stopplan)
+ {
+ stopplan = node->as_whichplan;
+ continue;
+ }
+
+ /* All subnodes are exhausted. Finish this node. */
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
+ return;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
@@ -277,6 +338,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index b72decc..b0a86c5 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1177,6 +1177,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
2.9.2
----Next_Part(Mon_Aug_29_17_08_36_2016_213)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Mon_Aug_29_17_08_36_2016_213)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 94 ++++++++++++++++++++++++++++++++-------
src/include/nodes/execnodes.h | 2 +
2 files changed, 80 insertions(+), 16 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..1c0d26e 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,49 +199,104 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
- for (;;)
+ int n_notready = 0;
+ PlanState *subnode;
+ int i, n;
+
+ n = node->as_whichplan;
+
+ for (i = 0 ; i < node->as_nplans ; i++)
{
- PlanState *subnode;
TupleTableSlot *result;
+ if (node->as_async)
+ {
+ if (n >= node->as_nplans)
+ n = 0;
+
+ if (node->as_finished[n])
+ {
+ n++;
+ continue;
+ }
+ }
+
/*
* figure out which subplan we are currently processing
*/
- subnode = node->appendplans[node->as_whichplan];
+ subnode = node->appendplans[n];
/*
- * get a tuple from the subplan
+ * execute the subplan to get a result if it is not ready yet
*/
- result = ExecProcNode(subnode);
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ /*
+ * This subnode is not ready yet when asynchrony is not allowed,
+ * immediately wait for this subnode.
+ */
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ n_notready++;
+ n++;
+ continue;
+ }
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ Assert(subnode->result_ready);
+
+ result = ExecConsumeResult((PlanState *)subnode);
if (!TupIsNull(result))
{
/*
- * If the subplan gave us something then return it as-is. We do
- * NOT make use of the result slot that was set up in
- * ExecInitAppend; there's no need for it.
+ * If the subplan gave us something then return it
+ * as-is. We do NOT make use of the result slot that was
+ * set up in ExecInitAppend; there's no need for it.
*/
+ node->as_whichplan = n;
ExecReturnTuple(&node->ps, result);
return;
}
+ /* Tuple has been exhausted on this subnode */
+ if (node->as_async)
+ {
+ node->as_finished[n++] = true;
+ continue;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
* ExecInitAppend.
*/
if (ScanDirectionIsForward(node->ps.state->es_direction))
- node->as_whichplan++;
+ {
+ if (n >= node->as_nplans - 1)
+ break;
+ n++;
+ }
else
- node->as_whichplan--;
- if (!exec_append_initialize_next(node))
{
- ExecReturnTuple(&node->ps,
- ExecClearTuple(node->ps.ps_ResultTupleSlot));
- return;
+ if (n == 0)
+ break;
+ n--;
}
+ }
- /* Else loop back and try to get a tuple from the new subplan */
+ /*
+ * We are finished if reached here and no subnodes are not-ready
+ */
+ if (n_notready == 0)
+ {
+ node->as_whichplan = n;
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
}
}
@@ -277,6 +337,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 9121537..e4f2bb6 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1170,6 +1170,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
1.8.3.1
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: Text/Plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline; filename="gentbl.sql"
CREATE EXTENSION postgres_fdw;
CREATE TABLE pl (a int, b int);
CREATE TABLE cl1 (LIKE pl) INHERITS (pl);
CREATE TABLE cl2 (LIKE pl) INHERITS (pl);
CREATE TABLE cl3 (LIKE pl) INHERITS (pl);
CREATE TABLE cl4 (LIKE pl) INHERITS (pl);
INSERT INTO cl1 (SELECT a, a FROM generate_series(0000000, 0999999) a);
INSERT INTO cl2 (SELECT a, a FROM generate_series(1000000, 1999999) a);
INSERT INTO cl3 (SELECT a, a FROM generate_series(2000000, 2999999) a);
INSERT INTO cl4 (SELECT a, a FROM generate_series(3000000, 3999999) a);
CREATE TABLE t0 (LIKE pl);
INSERT INTO t0 (SELECT a, a FROM generate_series(0000000, 9999999) a);
CREATE SERVER sv0 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv1 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv2 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv3 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv4 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE USER MAPPING FOR public SERVER sv0;
CREATE USER MAPPING FOR public SERVER sv1;
CREATE USER MAPPING FOR public SERVER sv2;
CREATE USER MAPPING FOR public SERVER sv3;
CREATE USER MAPPING FOR public SERVER sv4;
CREATE FOREIGN TABLE ft10 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl1');
CREATE FOREIGN TABLE ft20 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl2');
CREATE FOREIGN TABLE ft30 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl3');
CREATE FOREIGN TABLE ft40 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl4');
CREATE FOREIGN TABLE ft11 (a int, b int) SERVER sv1 OPTIONS (table_name 'cl1');
CREATE FOREIGN TABLE ft22 (a int, b int) SERVER sv2 OPTIONS (table_name 'cl2');
CREATE FOREIGN TABLE ft33 (a int, b int) SERVER sv3 OPTIONS (table_name 'cl3');
CREATE FOREIGN TABLE ft44 (a int, b int) SERVER sv4 OPTIONS (table_name 'cl4');
CREATE TABLE pf0 (LIKE pl);
ALTER FOREIGN TABLE ft10 INHERIT pf0;
ALTER FOREIGN TABLE ft20 INHERIT pf0;
ALTER FOREIGN TABLE ft30 INHERIT pf0;
ALTER FOREIGN TABLE ft40 INHERIT pf0;
CREATE table pf1 (LIKE pl);
ALTER FOREIGN TABLE ft11 INHERIT pf1;
ALTER FOREIGN TABLE ft22 INHERIT pf1;
ALTER FOREIGN TABLE ft33 INHERIT pf1;
ALTER FOREIGN TABLE ft44 INHERIT pf1;
ANALYZE;
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: Text/Plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline; filename="testrun.sh"
#! /bin/bash
function do_test() {
echo $1
for i in $(seq 1 10); do psql postgres -c "set log_min_duration_statement = 0; set client_min_messages=log; select sum(a) from $1"; done | grep LOG
}
for i in "t0" "pl" "pf0" "pf1";
do
do_test $i
done
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Wed_Jul_06_16_29_14_2016_599)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 94 ++++++++++++++++++++++++++++++++-------
src/include/nodes/execnodes.h | 2 +
2 files changed, 80 insertions(+), 16 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..1c0d26e 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,49 +199,104 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
- for (;;)
+ int n_notready = 0;
+ PlanState *subnode;
+ int i, n;
+
+ n = node->as_whichplan;
+
+ for (i = 0 ; i < node->as_nplans ; i++)
{
- PlanState *subnode;
TupleTableSlot *result;
+ if (node->as_async)
+ {
+ if (n >= node->as_nplans)
+ n = 0;
+
+ if (node->as_finished[n])
+ {
+ n++;
+ continue;
+ }
+ }
+
/*
* figure out which subplan we are currently processing
*/
- subnode = node->appendplans[node->as_whichplan];
+ subnode = node->appendplans[n];
/*
- * get a tuple from the subplan
+ * execute the subplan to get a result if it is not ready yet
*/
- result = ExecProcNode(subnode);
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ /*
+ * This subnode is not ready yet when asynchrony is not allowed,
+ * immediately wait for this subnode.
+ */
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ n_notready++;
+ n++;
+ continue;
+ }
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ Assert(subnode->result_ready);
+
+ result = ExecConsumeResult((PlanState *)subnode);
if (!TupIsNull(result))
{
/*
- * If the subplan gave us something then return it as-is. We do
- * NOT make use of the result slot that was set up in
- * ExecInitAppend; there's no need for it.
+ * If the subplan gave us something then return it
+ * as-is. We do NOT make use of the result slot that was
+ * set up in ExecInitAppend; there's no need for it.
*/
+ node->as_whichplan = n;
ExecReturnTuple(&node->ps, result);
return;
}
+ /* Tuple has been exhausted on this subnode */
+ if (node->as_async)
+ {
+ node->as_finished[n++] = true;
+ continue;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
* ExecInitAppend.
*/
if (ScanDirectionIsForward(node->ps.state->es_direction))
- node->as_whichplan++;
+ {
+ if (n >= node->as_nplans - 1)
+ break;
+ n++;
+ }
else
- node->as_whichplan--;
- if (!exec_append_initialize_next(node))
{
- ExecReturnTuple(&node->ps,
- ExecClearTuple(node->ps.ps_ResultTupleSlot));
- return;
+ if (n == 0)
+ break;
+ n--;
}
+ }
- /* Else loop back and try to get a tuple from the new subplan */
+ /*
+ * We are finished if reached here and no subnodes are not-ready
+ */
+ if (n_notready == 0)
+ {
+ node->as_whichplan = n;
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
}
}
@@ -277,6 +337,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 9121537..e4f2bb6 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1170,6 +1170,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
1.8.3.1
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: Text/Plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline; filename="gentbl.sql"
CREATE EXTENSION postgres_fdw;
CREATE TABLE pl (a int, b int);
CREATE TABLE cl1 (LIKE pl) INHERITS (pl);
CREATE TABLE cl2 (LIKE pl) INHERITS (pl);
CREATE TABLE cl3 (LIKE pl) INHERITS (pl);
CREATE TABLE cl4 (LIKE pl) INHERITS (pl);
INSERT INTO cl1 (SELECT a, a FROM generate_series(0000000, 0999999) a);
INSERT INTO cl2 (SELECT a, a FROM generate_series(1000000, 1999999) a);
INSERT INTO cl3 (SELECT a, a FROM generate_series(2000000, 2999999) a);
INSERT INTO cl4 (SELECT a, a FROM generate_series(3000000, 3999999) a);
CREATE TABLE t0 (LIKE pl);
INSERT INTO t0 (SELECT a, a FROM generate_series(0000000, 9999999) a);
CREATE SERVER sv0 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv1 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv2 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv3 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv4 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE USER MAPPING FOR public SERVER sv0;
CREATE USER MAPPING FOR public SERVER sv1;
CREATE USER MAPPING FOR public SERVER sv2;
CREATE USER MAPPING FOR public SERVER sv3;
CREATE USER MAPPING FOR public SERVER sv4;
CREATE FOREIGN TABLE ft10 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl1');
CREATE FOREIGN TABLE ft20 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl2');
CREATE FOREIGN TABLE ft30 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl3');
CREATE FOREIGN TABLE ft40 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl4');
CREATE FOREIGN TABLE ft11 (a int, b int) SERVER sv1 OPTIONS (table_name 'cl1');
CREATE FOREIGN TABLE ft22 (a int, b int) SERVER sv2 OPTIONS (table_name 'cl2');
CREATE FOREIGN TABLE ft33 (a int, b int) SERVER sv3 OPTIONS (table_name 'cl3');
CREATE FOREIGN TABLE ft44 (a int, b int) SERVER sv4 OPTIONS (table_name 'cl4');
CREATE TABLE pf0 (LIKE pl);
ALTER FOREIGN TABLE ft10 INHERIT pf0;
ALTER FOREIGN TABLE ft20 INHERIT pf0;
ALTER FOREIGN TABLE ft30 INHERIT pf0;
ALTER FOREIGN TABLE ft40 INHERIT pf0;
CREATE table pf1 (LIKE pl);
ALTER FOREIGN TABLE ft11 INHERIT pf1;
ALTER FOREIGN TABLE ft22 INHERIT pf1;
ALTER FOREIGN TABLE ft33 INHERIT pf1;
ALTER FOREIGN TABLE ft44 INHERIT pf1;
ANALYZE;
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: Text/Plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline; filename="testrun.sh"
#! /bin/bash
function do_test() {
echo $1
for i in $(seq 1 10); do psql postgres -c "set log_min_duration_statement = 0; set client_min_messages=log; select sum(a) from $1"; done | grep LOG
}
for i in "t0" "pl" "pf0" "pf1";
do
do_test $i
done
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Wed_Jul_06_16_29_14_2016_599)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 67 +++++++++++++++++++++++++++++++++++++--
src/include/nodes/execnodes.h | 2 ++
2 files changed, 67 insertions(+), 2 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..9a4063a 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,6 +199,8 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
+ int stopplan = node->as_whichplan;
+
for (;;)
{
PlanState *subnode;
@@ -207,7 +214,36 @@ ExecAppend(AppendState *node)
/*
* get a tuple from the subplan
*/
- result = ExecProcNode(subnode);
+ Assert(!node->as_finished[node->as_whichplan]);
+
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ /* Move to the next living node */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ /* No node is ready yet, return as not-ready */
+ if (node->as_whichplan == stopplan)
+ return;
+
+ /* Try the next node */
+ continue;
+ }
+
+ /* If not async, immediately wait for this subnode */
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ result = ExecConsumeResult((PlanState *) subnode);
if (!TupIsNull(result))
{
@@ -220,6 +256,31 @@ ExecAppend(AppendState *node)
return;
}
+ if (node->as_async)
+ {
+ node->as_finished[node->as_whichplan] = true;
+ stopplan = node->as_whichplan;
+
+ /* Find the next living subnode */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ if (node->as_whichplan != stopplan)
+ {
+ stopplan = node->as_whichplan;
+ continue;
+ }
+
+ /* All subnodes are exhausted. Finish this node. */
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
+ return;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
@@ -277,6 +338,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index b72decc..b0a86c5 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1177,6 +1177,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
1.8.3.1
----Next_Part(Thu_Jul_21_18_50_07_2016_597)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Thu_Jul_21_18_50_07_2016_597)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 67 +++++++++++++++++++++++++++++++++++++--
src/include/nodes/execnodes.h | 2 ++
2 files changed, 67 insertions(+), 2 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..9a4063a 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,6 +199,8 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
+ int stopplan = node->as_whichplan;
+
for (;;)
{
PlanState *subnode;
@@ -207,7 +214,36 @@ ExecAppend(AppendState *node)
/*
* get a tuple from the subplan
*/
- result = ExecProcNode(subnode);
+ Assert(!node->as_finished[node->as_whichplan]);
+
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ /* Move to the next living node */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ /* No node is ready yet, return as not-ready */
+ if (node->as_whichplan == stopplan)
+ return;
+
+ /* Try the next node */
+ continue;
+ }
+
+ /* If not async, immediately wait for this subnode */
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ result = ExecConsumeResult((PlanState *) subnode);
if (!TupIsNull(result))
{
@@ -220,6 +256,31 @@ ExecAppend(AppendState *node)
return;
}
+ if (node->as_async)
+ {
+ node->as_finished[node->as_whichplan] = true;
+ stopplan = node->as_whichplan;
+
+ /* Find the next living subnode */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ if (node->as_whichplan != stopplan)
+ {
+ stopplan = node->as_whichplan;
+ continue;
+ }
+
+ /* All subnodes are exhausted. Finish this node. */
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
+ return;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
@@ -277,6 +338,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index b72decc..b0a86c5 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1177,6 +1177,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
2.9.2
----Next_Part(Mon_Aug_29_17_08_36_2016_213)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Mon_Aug_29_17_08_36_2016_213)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 67 +++++++++++++++++++++++++++++++++++++--
src/include/nodes/execnodes.h | 2 ++
2 files changed, 67 insertions(+), 2 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..9a4063a 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,6 +199,8 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
+ int stopplan = node->as_whichplan;
+
for (;;)
{
PlanState *subnode;
@@ -207,7 +214,36 @@ ExecAppend(AppendState *node)
/*
* get a tuple from the subplan
*/
- result = ExecProcNode(subnode);
+ Assert(!node->as_finished[node->as_whichplan]);
+
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ /* Move to the next living node */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ /* No node is ready yet, return as not-ready */
+ if (node->as_whichplan == stopplan)
+ return;
+
+ /* Try the next node */
+ continue;
+ }
+
+ /* If not async, immediately wait for this subnode */
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ result = ExecConsumeResult((PlanState *) subnode);
if (!TupIsNull(result))
{
@@ -220,6 +256,31 @@ ExecAppend(AppendState *node)
return;
}
+ if (node->as_async)
+ {
+ node->as_finished[node->as_whichplan] = true;
+ stopplan = node->as_whichplan;
+
+ /* Find the next living subnode */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ if (node->as_whichplan != stopplan)
+ {
+ stopplan = node->as_whichplan;
+ continue;
+ }
+
+ /* All subnodes are exhausted. Finish this node. */
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
+ return;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
@@ -277,6 +338,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index b72decc..b0a86c5 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1177,6 +1177,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
2.9.2
----Next_Part(Mon_Aug_29_17_08_36_2016_213)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Mon_Aug_29_17_08_36_2016_213)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 67 +++++++++++++++++++++++++++++++++++++--
src/include/nodes/execnodes.h | 2 ++
2 files changed, 67 insertions(+), 2 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..9a4063a 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,6 +199,8 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
+ int stopplan = node->as_whichplan;
+
for (;;)
{
PlanState *subnode;
@@ -207,7 +214,36 @@ ExecAppend(AppendState *node)
/*
* get a tuple from the subplan
*/
- result = ExecProcNode(subnode);
+ Assert(!node->as_finished[node->as_whichplan]);
+
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ /* Move to the next living node */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ /* No node is ready yet, return as not-ready */
+ if (node->as_whichplan == stopplan)
+ return;
+
+ /* Try the next node */
+ continue;
+ }
+
+ /* If not async, immediately wait for this subnode */
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ result = ExecConsumeResult((PlanState *) subnode);
if (!TupIsNull(result))
{
@@ -220,6 +256,31 @@ ExecAppend(AppendState *node)
return;
}
+ if (node->as_async)
+ {
+ node->as_finished[node->as_whichplan] = true;
+ stopplan = node->as_whichplan;
+
+ /* Find the next living subnode */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ if (node->as_whichplan != stopplan)
+ {
+ stopplan = node->as_whichplan;
+ continue;
+ }
+
+ /* All subnodes are exhausted. Finish this node. */
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
+ return;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
@@ -277,6 +338,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index b72decc..b0a86c5 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1177,6 +1177,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
1.8.3.1
----Next_Part(Thu_Jul_21_18_50_07_2016_597)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Thu_Jul_21_18_50_07_2016_597)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 67 +++++++++++++++++++++++++++++++++++++--
src/include/nodes/execnodes.h | 2 ++
2 files changed, 67 insertions(+), 2 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..9a4063a 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,6 +199,8 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
+ int stopplan = node->as_whichplan;
+
for (;;)
{
PlanState *subnode;
@@ -207,7 +214,36 @@ ExecAppend(AppendState *node)
/*
* get a tuple from the subplan
*/
- result = ExecProcNode(subnode);
+ Assert(!node->as_finished[node->as_whichplan]);
+
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ /* Move to the next living node */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ /* No node is ready yet, return as not-ready */
+ if (node->as_whichplan == stopplan)
+ return;
+
+ /* Try the next node */
+ continue;
+ }
+
+ /* If not async, immediately wait for this subnode */
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ result = ExecConsumeResult((PlanState *) subnode);
if (!TupIsNull(result))
{
@@ -220,6 +256,31 @@ ExecAppend(AppendState *node)
return;
}
+ if (node->as_async)
+ {
+ node->as_finished[node->as_whichplan] = true;
+ stopplan = node->as_whichplan;
+
+ /* Find the next living subnode */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ if (node->as_whichplan != stopplan)
+ {
+ stopplan = node->as_whichplan;
+ continue;
+ }
+
+ /* All subnodes are exhausted. Finish this node. */
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
+ return;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
@@ -277,6 +338,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index b72decc..b0a86c5 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1177,6 +1177,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
1.8.3.1
----Next_Part(Thu_Jul_21_18_50_07_2016_597)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Thu_Jul_21_18_50_07_2016_597)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 67 +++++++++++++++++++++++++++++++++++++--
src/include/nodes/execnodes.h | 2 ++
2 files changed, 67 insertions(+), 2 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..9a4063a 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,6 +199,8 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
+ int stopplan = node->as_whichplan;
+
for (;;)
{
PlanState *subnode;
@@ -207,7 +214,36 @@ ExecAppend(AppendState *node)
/*
* get a tuple from the subplan
*/
- result = ExecProcNode(subnode);
+ Assert(!node->as_finished[node->as_whichplan]);
+
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ /* Move to the next living node */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ /* No node is ready yet, return as not-ready */
+ if (node->as_whichplan == stopplan)
+ return;
+
+ /* Try the next node */
+ continue;
+ }
+
+ /* If not async, immediately wait for this subnode */
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ result = ExecConsumeResult((PlanState *) subnode);
if (!TupIsNull(result))
{
@@ -220,6 +256,31 @@ ExecAppend(AppendState *node)
return;
}
+ if (node->as_async)
+ {
+ node->as_finished[node->as_whichplan] = true;
+ stopplan = node->as_whichplan;
+
+ /* Find the next living subnode */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ if (node->as_whichplan != stopplan)
+ {
+ stopplan = node->as_whichplan;
+ continue;
+ }
+
+ /* All subnodes are exhausted. Finish this node. */
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
+ return;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
@@ -277,6 +338,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index b72decc..b0a86c5 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1177,6 +1177,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
2.9.2
----Next_Part(Mon_Aug_29_17_08_36_2016_213)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Mon_Aug_29_17_08_36_2016_213)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 67 +++++++++++++++++++++++++++++++++++++--
src/include/nodes/execnodes.h | 2 ++
2 files changed, 67 insertions(+), 2 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..9a4063a 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,6 +199,8 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
+ int stopplan = node->as_whichplan;
+
for (;;)
{
PlanState *subnode;
@@ -207,7 +214,36 @@ ExecAppend(AppendState *node)
/*
* get a tuple from the subplan
*/
- result = ExecProcNode(subnode);
+ Assert(!node->as_finished[node->as_whichplan]);
+
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ /* Move to the next living node */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ /* No node is ready yet, return as not-ready */
+ if (node->as_whichplan == stopplan)
+ return;
+
+ /* Try the next node */
+ continue;
+ }
+
+ /* If not async, immediately wait for this subnode */
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ result = ExecConsumeResult((PlanState *) subnode);
if (!TupIsNull(result))
{
@@ -220,6 +256,31 @@ ExecAppend(AppendState *node)
return;
}
+ if (node->as_async)
+ {
+ node->as_finished[node->as_whichplan] = true;
+ stopplan = node->as_whichplan;
+
+ /* Find the next living subnode */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ if (node->as_whichplan != stopplan)
+ {
+ stopplan = node->as_whichplan;
+ continue;
+ }
+
+ /* All subnodes are exhausted. Finish this node. */
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
+ return;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
@@ -277,6 +338,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index b72decc..b0a86c5 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1177,6 +1177,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
2.9.2
----Next_Part(Mon_Aug_29_17_08_36_2016_213)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Mon_Aug_29_17_08_36_2016_213)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 94 ++++++++++++++++++++++++++++++++-------
src/include/nodes/execnodes.h | 2 +
2 files changed, 80 insertions(+), 16 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..1c0d26e 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,49 +199,104 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
- for (;;)
+ int n_notready = 0;
+ PlanState *subnode;
+ int i, n;
+
+ n = node->as_whichplan;
+
+ for (i = 0 ; i < node->as_nplans ; i++)
{
- PlanState *subnode;
TupleTableSlot *result;
+ if (node->as_async)
+ {
+ if (n >= node->as_nplans)
+ n = 0;
+
+ if (node->as_finished[n])
+ {
+ n++;
+ continue;
+ }
+ }
+
/*
* figure out which subplan we are currently processing
*/
- subnode = node->appendplans[node->as_whichplan];
+ subnode = node->appendplans[n];
/*
- * get a tuple from the subplan
+ * execute the subplan to get a result if it is not ready yet
*/
- result = ExecProcNode(subnode);
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ /*
+ * This subnode is not ready yet when asynchrony is not allowed,
+ * immediately wait for this subnode.
+ */
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ n_notready++;
+ n++;
+ continue;
+ }
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ Assert(subnode->result_ready);
+
+ result = ExecConsumeResult((PlanState *)subnode);
if (!TupIsNull(result))
{
/*
- * If the subplan gave us something then return it as-is. We do
- * NOT make use of the result slot that was set up in
- * ExecInitAppend; there's no need for it.
+ * If the subplan gave us something then return it
+ * as-is. We do NOT make use of the result slot that was
+ * set up in ExecInitAppend; there's no need for it.
*/
+ node->as_whichplan = n;
ExecReturnTuple(&node->ps, result);
return;
}
+ /* Tuple has been exhausted on this subnode */
+ if (node->as_async)
+ {
+ node->as_finished[n++] = true;
+ continue;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
* ExecInitAppend.
*/
if (ScanDirectionIsForward(node->ps.state->es_direction))
- node->as_whichplan++;
+ {
+ if (n >= node->as_nplans - 1)
+ break;
+ n++;
+ }
else
- node->as_whichplan--;
- if (!exec_append_initialize_next(node))
{
- ExecReturnTuple(&node->ps,
- ExecClearTuple(node->ps.ps_ResultTupleSlot));
- return;
+ if (n == 0)
+ break;
+ n--;
}
+ }
- /* Else loop back and try to get a tuple from the new subplan */
+ /*
+ * We are finished if reached here and no subnodes are not-ready
+ */
+ if (n_notready == 0)
+ {
+ node->as_whichplan = n;
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
}
}
@@ -277,6 +337,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 9121537..e4f2bb6 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1170,6 +1170,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
1.8.3.1
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: Text/Plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline; filename="gentbl.sql"
CREATE EXTENSION postgres_fdw;
CREATE TABLE pl (a int, b int);
CREATE TABLE cl1 (LIKE pl) INHERITS (pl);
CREATE TABLE cl2 (LIKE pl) INHERITS (pl);
CREATE TABLE cl3 (LIKE pl) INHERITS (pl);
CREATE TABLE cl4 (LIKE pl) INHERITS (pl);
INSERT INTO cl1 (SELECT a, a FROM generate_series(0000000, 0999999) a);
INSERT INTO cl2 (SELECT a, a FROM generate_series(1000000, 1999999) a);
INSERT INTO cl3 (SELECT a, a FROM generate_series(2000000, 2999999) a);
INSERT INTO cl4 (SELECT a, a FROM generate_series(3000000, 3999999) a);
CREATE TABLE t0 (LIKE pl);
INSERT INTO t0 (SELECT a, a FROM generate_series(0000000, 9999999) a);
CREATE SERVER sv0 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv1 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv2 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv3 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv4 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE USER MAPPING FOR public SERVER sv0;
CREATE USER MAPPING FOR public SERVER sv1;
CREATE USER MAPPING FOR public SERVER sv2;
CREATE USER MAPPING FOR public SERVER sv3;
CREATE USER MAPPING FOR public SERVER sv4;
CREATE FOREIGN TABLE ft10 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl1');
CREATE FOREIGN TABLE ft20 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl2');
CREATE FOREIGN TABLE ft30 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl3');
CREATE FOREIGN TABLE ft40 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl4');
CREATE FOREIGN TABLE ft11 (a int, b int) SERVER sv1 OPTIONS (table_name 'cl1');
CREATE FOREIGN TABLE ft22 (a int, b int) SERVER sv2 OPTIONS (table_name 'cl2');
CREATE FOREIGN TABLE ft33 (a int, b int) SERVER sv3 OPTIONS (table_name 'cl3');
CREATE FOREIGN TABLE ft44 (a int, b int) SERVER sv4 OPTIONS (table_name 'cl4');
CREATE TABLE pf0 (LIKE pl);
ALTER FOREIGN TABLE ft10 INHERIT pf0;
ALTER FOREIGN TABLE ft20 INHERIT pf0;
ALTER FOREIGN TABLE ft30 INHERIT pf0;
ALTER FOREIGN TABLE ft40 INHERIT pf0;
CREATE table pf1 (LIKE pl);
ALTER FOREIGN TABLE ft11 INHERIT pf1;
ALTER FOREIGN TABLE ft22 INHERIT pf1;
ALTER FOREIGN TABLE ft33 INHERIT pf1;
ALTER FOREIGN TABLE ft44 INHERIT pf1;
ANALYZE;
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: Text/Plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline; filename="testrun.sh"
#! /bin/bash
function do_test() {
echo $1
for i in $(seq 1 10); do psql postgres -c "set log_min_duration_statement = 0; set client_min_messages=log; select sum(a) from $1"; done | grep LOG
}
for i in "t0" "pl" "pf0" "pf1";
do
do_test $i
done
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Wed_Jul_06_16_29_14_2016_599)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 94 ++++++++++++++++++++++++++++++++-------
src/include/nodes/execnodes.h | 2 +
2 files changed, 80 insertions(+), 16 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..1c0d26e 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,49 +199,104 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
- for (;;)
+ int n_notready = 0;
+ PlanState *subnode;
+ int i, n;
+
+ n = node->as_whichplan;
+
+ for (i = 0 ; i < node->as_nplans ; i++)
{
- PlanState *subnode;
TupleTableSlot *result;
+ if (node->as_async)
+ {
+ if (n >= node->as_nplans)
+ n = 0;
+
+ if (node->as_finished[n])
+ {
+ n++;
+ continue;
+ }
+ }
+
/*
* figure out which subplan we are currently processing
*/
- subnode = node->appendplans[node->as_whichplan];
+ subnode = node->appendplans[n];
/*
- * get a tuple from the subplan
+ * execute the subplan to get a result if it is not ready yet
*/
- result = ExecProcNode(subnode);
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ /*
+ * This subnode is not ready yet when asynchrony is not allowed,
+ * immediately wait for this subnode.
+ */
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ n_notready++;
+ n++;
+ continue;
+ }
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ Assert(subnode->result_ready);
+
+ result = ExecConsumeResult((PlanState *)subnode);
if (!TupIsNull(result))
{
/*
- * If the subplan gave us something then return it as-is. We do
- * NOT make use of the result slot that was set up in
- * ExecInitAppend; there's no need for it.
+ * If the subplan gave us something then return it
+ * as-is. We do NOT make use of the result slot that was
+ * set up in ExecInitAppend; there's no need for it.
*/
+ node->as_whichplan = n;
ExecReturnTuple(&node->ps, result);
return;
}
+ /* Tuple has been exhausted on this subnode */
+ if (node->as_async)
+ {
+ node->as_finished[n++] = true;
+ continue;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
* ExecInitAppend.
*/
if (ScanDirectionIsForward(node->ps.state->es_direction))
- node->as_whichplan++;
+ {
+ if (n >= node->as_nplans - 1)
+ break;
+ n++;
+ }
else
- node->as_whichplan--;
- if (!exec_append_initialize_next(node))
{
- ExecReturnTuple(&node->ps,
- ExecClearTuple(node->ps.ps_ResultTupleSlot));
- return;
+ if (n == 0)
+ break;
+ n--;
}
+ }
- /* Else loop back and try to get a tuple from the new subplan */
+ /*
+ * We are finished if reached here and no subnodes are not-ready
+ */
+ if (n_notready == 0)
+ {
+ node->as_whichplan = n;
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
}
}
@@ -277,6 +337,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 9121537..e4f2bb6 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1170,6 +1170,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
1.8.3.1
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: Text/Plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline; filename="gentbl.sql"
CREATE EXTENSION postgres_fdw;
CREATE TABLE pl (a int, b int);
CREATE TABLE cl1 (LIKE pl) INHERITS (pl);
CREATE TABLE cl2 (LIKE pl) INHERITS (pl);
CREATE TABLE cl3 (LIKE pl) INHERITS (pl);
CREATE TABLE cl4 (LIKE pl) INHERITS (pl);
INSERT INTO cl1 (SELECT a, a FROM generate_series(0000000, 0999999) a);
INSERT INTO cl2 (SELECT a, a FROM generate_series(1000000, 1999999) a);
INSERT INTO cl3 (SELECT a, a FROM generate_series(2000000, 2999999) a);
INSERT INTO cl4 (SELECT a, a FROM generate_series(3000000, 3999999) a);
CREATE TABLE t0 (LIKE pl);
INSERT INTO t0 (SELECT a, a FROM generate_series(0000000, 9999999) a);
CREATE SERVER sv0 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv1 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv2 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv3 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv4 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE USER MAPPING FOR public SERVER sv0;
CREATE USER MAPPING FOR public SERVER sv1;
CREATE USER MAPPING FOR public SERVER sv2;
CREATE USER MAPPING FOR public SERVER sv3;
CREATE USER MAPPING FOR public SERVER sv4;
CREATE FOREIGN TABLE ft10 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl1');
CREATE FOREIGN TABLE ft20 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl2');
CREATE FOREIGN TABLE ft30 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl3');
CREATE FOREIGN TABLE ft40 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl4');
CREATE FOREIGN TABLE ft11 (a int, b int) SERVER sv1 OPTIONS (table_name 'cl1');
CREATE FOREIGN TABLE ft22 (a int, b int) SERVER sv2 OPTIONS (table_name 'cl2');
CREATE FOREIGN TABLE ft33 (a int, b int) SERVER sv3 OPTIONS (table_name 'cl3');
CREATE FOREIGN TABLE ft44 (a int, b int) SERVER sv4 OPTIONS (table_name 'cl4');
CREATE TABLE pf0 (LIKE pl);
ALTER FOREIGN TABLE ft10 INHERIT pf0;
ALTER FOREIGN TABLE ft20 INHERIT pf0;
ALTER FOREIGN TABLE ft30 INHERIT pf0;
ALTER FOREIGN TABLE ft40 INHERIT pf0;
CREATE table pf1 (LIKE pl);
ALTER FOREIGN TABLE ft11 INHERIT pf1;
ALTER FOREIGN TABLE ft22 INHERIT pf1;
ALTER FOREIGN TABLE ft33 INHERIT pf1;
ALTER FOREIGN TABLE ft44 INHERIT pf1;
ANALYZE;
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: Text/Plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline; filename="testrun.sh"
#! /bin/bash
function do_test() {
echo $1
for i in $(seq 1 10); do psql postgres -c "set log_min_duration_statement = 0; set client_min_messages=log; select sum(a) from $1"; done | grep LOG
}
for i in "t0" "pl" "pf0" "pf1";
do
do_test $i
done
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Wed_Jul_06_16_29_14_2016_599)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 67 +++++++++++++++++++++++++++++++++++++--
src/include/nodes/execnodes.h | 2 ++
2 files changed, 67 insertions(+), 2 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..9a4063a 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,6 +199,8 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
+ int stopplan = node->as_whichplan;
+
for (;;)
{
PlanState *subnode;
@@ -207,7 +214,36 @@ ExecAppend(AppendState *node)
/*
* get a tuple from the subplan
*/
- result = ExecProcNode(subnode);
+ Assert(!node->as_finished[node->as_whichplan]);
+
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ /* Move to the next living node */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ /* No node is ready yet, return as not-ready */
+ if (node->as_whichplan == stopplan)
+ return;
+
+ /* Try the next node */
+ continue;
+ }
+
+ /* If not async, immediately wait for this subnode */
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ result = ExecConsumeResult((PlanState *) subnode);
if (!TupIsNull(result))
{
@@ -220,6 +256,31 @@ ExecAppend(AppendState *node)
return;
}
+ if (node->as_async)
+ {
+ node->as_finished[node->as_whichplan] = true;
+ stopplan = node->as_whichplan;
+
+ /* Find the next living subnode */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ if (node->as_whichplan != stopplan)
+ {
+ stopplan = node->as_whichplan;
+ continue;
+ }
+
+ /* All subnodes are exhausted. Finish this node. */
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
+ return;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
@@ -277,6 +338,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index b72decc..b0a86c5 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1177,6 +1177,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
1.8.3.1
----Next_Part(Thu_Jul_21_18_50_07_2016_597)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Thu_Jul_21_18_50_07_2016_597)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 94 ++++++++++++++++++++++++++++++++-------
src/include/nodes/execnodes.h | 2 +
2 files changed, 80 insertions(+), 16 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..1c0d26e 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,49 +199,104 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
- for (;;)
+ int n_notready = 0;
+ PlanState *subnode;
+ int i, n;
+
+ n = node->as_whichplan;
+
+ for (i = 0 ; i < node->as_nplans ; i++)
{
- PlanState *subnode;
TupleTableSlot *result;
+ if (node->as_async)
+ {
+ if (n >= node->as_nplans)
+ n = 0;
+
+ if (node->as_finished[n])
+ {
+ n++;
+ continue;
+ }
+ }
+
/*
* figure out which subplan we are currently processing
*/
- subnode = node->appendplans[node->as_whichplan];
+ subnode = node->appendplans[n];
/*
- * get a tuple from the subplan
+ * execute the subplan to get a result if it is not ready yet
*/
- result = ExecProcNode(subnode);
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ /*
+ * This subnode is not ready yet when asynchrony is not allowed,
+ * immediately wait for this subnode.
+ */
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ n_notready++;
+ n++;
+ continue;
+ }
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ Assert(subnode->result_ready);
+
+ result = ExecConsumeResult((PlanState *)subnode);
if (!TupIsNull(result))
{
/*
- * If the subplan gave us something then return it as-is. We do
- * NOT make use of the result slot that was set up in
- * ExecInitAppend; there's no need for it.
+ * If the subplan gave us something then return it
+ * as-is. We do NOT make use of the result slot that was
+ * set up in ExecInitAppend; there's no need for it.
*/
+ node->as_whichplan = n;
ExecReturnTuple(&node->ps, result);
return;
}
+ /* Tuple has been exhausted on this subnode */
+ if (node->as_async)
+ {
+ node->as_finished[n++] = true;
+ continue;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
* ExecInitAppend.
*/
if (ScanDirectionIsForward(node->ps.state->es_direction))
- node->as_whichplan++;
+ {
+ if (n >= node->as_nplans - 1)
+ break;
+ n++;
+ }
else
- node->as_whichplan--;
- if (!exec_append_initialize_next(node))
{
- ExecReturnTuple(&node->ps,
- ExecClearTuple(node->ps.ps_ResultTupleSlot));
- return;
+ if (n == 0)
+ break;
+ n--;
}
+ }
- /* Else loop back and try to get a tuple from the new subplan */
+ /*
+ * We are finished if reached here and no subnodes are not-ready
+ */
+ if (n_notready == 0)
+ {
+ node->as_whichplan = n;
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
}
}
@@ -277,6 +337,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 9121537..e4f2bb6 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1170,6 +1170,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
1.8.3.1
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: Text/Plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline; filename="gentbl.sql"
CREATE EXTENSION postgres_fdw;
CREATE TABLE pl (a int, b int);
CREATE TABLE cl1 (LIKE pl) INHERITS (pl);
CREATE TABLE cl2 (LIKE pl) INHERITS (pl);
CREATE TABLE cl3 (LIKE pl) INHERITS (pl);
CREATE TABLE cl4 (LIKE pl) INHERITS (pl);
INSERT INTO cl1 (SELECT a, a FROM generate_series(0000000, 0999999) a);
INSERT INTO cl2 (SELECT a, a FROM generate_series(1000000, 1999999) a);
INSERT INTO cl3 (SELECT a, a FROM generate_series(2000000, 2999999) a);
INSERT INTO cl4 (SELECT a, a FROM generate_series(3000000, 3999999) a);
CREATE TABLE t0 (LIKE pl);
INSERT INTO t0 (SELECT a, a FROM generate_series(0000000, 9999999) a);
CREATE SERVER sv0 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv1 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv2 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv3 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv4 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE USER MAPPING FOR public SERVER sv0;
CREATE USER MAPPING FOR public SERVER sv1;
CREATE USER MAPPING FOR public SERVER sv2;
CREATE USER MAPPING FOR public SERVER sv3;
CREATE USER MAPPING FOR public SERVER sv4;
CREATE FOREIGN TABLE ft10 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl1');
CREATE FOREIGN TABLE ft20 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl2');
CREATE FOREIGN TABLE ft30 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl3');
CREATE FOREIGN TABLE ft40 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl4');
CREATE FOREIGN TABLE ft11 (a int, b int) SERVER sv1 OPTIONS (table_name 'cl1');
CREATE FOREIGN TABLE ft22 (a int, b int) SERVER sv2 OPTIONS (table_name 'cl2');
CREATE FOREIGN TABLE ft33 (a int, b int) SERVER sv3 OPTIONS (table_name 'cl3');
CREATE FOREIGN TABLE ft44 (a int, b int) SERVER sv4 OPTIONS (table_name 'cl4');
CREATE TABLE pf0 (LIKE pl);
ALTER FOREIGN TABLE ft10 INHERIT pf0;
ALTER FOREIGN TABLE ft20 INHERIT pf0;
ALTER FOREIGN TABLE ft30 INHERIT pf0;
ALTER FOREIGN TABLE ft40 INHERIT pf0;
CREATE table pf1 (LIKE pl);
ALTER FOREIGN TABLE ft11 INHERIT pf1;
ALTER FOREIGN TABLE ft22 INHERIT pf1;
ALTER FOREIGN TABLE ft33 INHERIT pf1;
ALTER FOREIGN TABLE ft44 INHERIT pf1;
ANALYZE;
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: Text/Plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline; filename="testrun.sh"
#! /bin/bash
function do_test() {
echo $1
for i in $(seq 1 10); do psql postgres -c "set log_min_duration_statement = 0; set client_min_messages=log; select sum(a) from $1"; done | grep LOG
}
for i in "t0" "pl" "pf0" "pf1";
do
do_test $i
done
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Wed_Jul_06_16_29_14_2016_599)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 94 ++++++++++++++++++++++++++++++++-------
src/include/nodes/execnodes.h | 2 +
2 files changed, 80 insertions(+), 16 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..1c0d26e 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,49 +199,104 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
- for (;;)
+ int n_notready = 0;
+ PlanState *subnode;
+ int i, n;
+
+ n = node->as_whichplan;
+
+ for (i = 0 ; i < node->as_nplans ; i++)
{
- PlanState *subnode;
TupleTableSlot *result;
+ if (node->as_async)
+ {
+ if (n >= node->as_nplans)
+ n = 0;
+
+ if (node->as_finished[n])
+ {
+ n++;
+ continue;
+ }
+ }
+
/*
* figure out which subplan we are currently processing
*/
- subnode = node->appendplans[node->as_whichplan];
+ subnode = node->appendplans[n];
/*
- * get a tuple from the subplan
+ * execute the subplan to get a result if it is not ready yet
*/
- result = ExecProcNode(subnode);
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ /*
+ * This subnode is not ready yet when asynchrony is not allowed,
+ * immediately wait for this subnode.
+ */
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ n_notready++;
+ n++;
+ continue;
+ }
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ Assert(subnode->result_ready);
+
+ result = ExecConsumeResult((PlanState *)subnode);
if (!TupIsNull(result))
{
/*
- * If the subplan gave us something then return it as-is. We do
- * NOT make use of the result slot that was set up in
- * ExecInitAppend; there's no need for it.
+ * If the subplan gave us something then return it
+ * as-is. We do NOT make use of the result slot that was
+ * set up in ExecInitAppend; there's no need for it.
*/
+ node->as_whichplan = n;
ExecReturnTuple(&node->ps, result);
return;
}
+ /* Tuple has been exhausted on this subnode */
+ if (node->as_async)
+ {
+ node->as_finished[n++] = true;
+ continue;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
* ExecInitAppend.
*/
if (ScanDirectionIsForward(node->ps.state->es_direction))
- node->as_whichplan++;
+ {
+ if (n >= node->as_nplans - 1)
+ break;
+ n++;
+ }
else
- node->as_whichplan--;
- if (!exec_append_initialize_next(node))
{
- ExecReturnTuple(&node->ps,
- ExecClearTuple(node->ps.ps_ResultTupleSlot));
- return;
+ if (n == 0)
+ break;
+ n--;
}
+ }
- /* Else loop back and try to get a tuple from the new subplan */
+ /*
+ * We are finished if reached here and no subnodes are not-ready
+ */
+ if (n_notready == 0)
+ {
+ node->as_whichplan = n;
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
}
}
@@ -277,6 +337,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 9121537..e4f2bb6 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1170,6 +1170,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
1.8.3.1
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: Text/Plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline; filename="gentbl.sql"
CREATE EXTENSION postgres_fdw;
CREATE TABLE pl (a int, b int);
CREATE TABLE cl1 (LIKE pl) INHERITS (pl);
CREATE TABLE cl2 (LIKE pl) INHERITS (pl);
CREATE TABLE cl3 (LIKE pl) INHERITS (pl);
CREATE TABLE cl4 (LIKE pl) INHERITS (pl);
INSERT INTO cl1 (SELECT a, a FROM generate_series(0000000, 0999999) a);
INSERT INTO cl2 (SELECT a, a FROM generate_series(1000000, 1999999) a);
INSERT INTO cl3 (SELECT a, a FROM generate_series(2000000, 2999999) a);
INSERT INTO cl4 (SELECT a, a FROM generate_series(3000000, 3999999) a);
CREATE TABLE t0 (LIKE pl);
INSERT INTO t0 (SELECT a, a FROM generate_series(0000000, 9999999) a);
CREATE SERVER sv0 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv1 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv2 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv3 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv4 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE USER MAPPING FOR public SERVER sv0;
CREATE USER MAPPING FOR public SERVER sv1;
CREATE USER MAPPING FOR public SERVER sv2;
CREATE USER MAPPING FOR public SERVER sv3;
CREATE USER MAPPING FOR public SERVER sv4;
CREATE FOREIGN TABLE ft10 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl1');
CREATE FOREIGN TABLE ft20 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl2');
CREATE FOREIGN TABLE ft30 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl3');
CREATE FOREIGN TABLE ft40 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl4');
CREATE FOREIGN TABLE ft11 (a int, b int) SERVER sv1 OPTIONS (table_name 'cl1');
CREATE FOREIGN TABLE ft22 (a int, b int) SERVER sv2 OPTIONS (table_name 'cl2');
CREATE FOREIGN TABLE ft33 (a int, b int) SERVER sv3 OPTIONS (table_name 'cl3');
CREATE FOREIGN TABLE ft44 (a int, b int) SERVER sv4 OPTIONS (table_name 'cl4');
CREATE TABLE pf0 (LIKE pl);
ALTER FOREIGN TABLE ft10 INHERIT pf0;
ALTER FOREIGN TABLE ft20 INHERIT pf0;
ALTER FOREIGN TABLE ft30 INHERIT pf0;
ALTER FOREIGN TABLE ft40 INHERIT pf0;
CREATE table pf1 (LIKE pl);
ALTER FOREIGN TABLE ft11 INHERIT pf1;
ALTER FOREIGN TABLE ft22 INHERIT pf1;
ALTER FOREIGN TABLE ft33 INHERIT pf1;
ALTER FOREIGN TABLE ft44 INHERIT pf1;
ANALYZE;
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: Text/Plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline; filename="testrun.sh"
#! /bin/bash
function do_test() {
echo $1
for i in $(seq 1 10); do psql postgres -c "set log_min_duration_statement = 0; set client_min_messages=log; select sum(a) from $1"; done | grep LOG
}
for i in "t0" "pl" "pf0" "pf1";
do
do_test $i
done
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Wed_Jul_06_16_29_14_2016_599)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 94 ++++++++++++++++++++++++++++++++-------
src/include/nodes/execnodes.h | 2 +
2 files changed, 80 insertions(+), 16 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..1c0d26e 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,49 +199,104 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
- for (;;)
+ int n_notready = 0;
+ PlanState *subnode;
+ int i, n;
+
+ n = node->as_whichplan;
+
+ for (i = 0 ; i < node->as_nplans ; i++)
{
- PlanState *subnode;
TupleTableSlot *result;
+ if (node->as_async)
+ {
+ if (n >= node->as_nplans)
+ n = 0;
+
+ if (node->as_finished[n])
+ {
+ n++;
+ continue;
+ }
+ }
+
/*
* figure out which subplan we are currently processing
*/
- subnode = node->appendplans[node->as_whichplan];
+ subnode = node->appendplans[n];
/*
- * get a tuple from the subplan
+ * execute the subplan to get a result if it is not ready yet
*/
- result = ExecProcNode(subnode);
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ /*
+ * This subnode is not ready yet when asynchrony is not allowed,
+ * immediately wait for this subnode.
+ */
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ n_notready++;
+ n++;
+ continue;
+ }
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ Assert(subnode->result_ready);
+
+ result = ExecConsumeResult((PlanState *)subnode);
if (!TupIsNull(result))
{
/*
- * If the subplan gave us something then return it as-is. We do
- * NOT make use of the result slot that was set up in
- * ExecInitAppend; there's no need for it.
+ * If the subplan gave us something then return it
+ * as-is. We do NOT make use of the result slot that was
+ * set up in ExecInitAppend; there's no need for it.
*/
+ node->as_whichplan = n;
ExecReturnTuple(&node->ps, result);
return;
}
+ /* Tuple has been exhausted on this subnode */
+ if (node->as_async)
+ {
+ node->as_finished[n++] = true;
+ continue;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
* ExecInitAppend.
*/
if (ScanDirectionIsForward(node->ps.state->es_direction))
- node->as_whichplan++;
+ {
+ if (n >= node->as_nplans - 1)
+ break;
+ n++;
+ }
else
- node->as_whichplan--;
- if (!exec_append_initialize_next(node))
{
- ExecReturnTuple(&node->ps,
- ExecClearTuple(node->ps.ps_ResultTupleSlot));
- return;
+ if (n == 0)
+ break;
+ n--;
}
+ }
- /* Else loop back and try to get a tuple from the new subplan */
+ /*
+ * We are finished if reached here and no subnodes are not-ready
+ */
+ if (n_notready == 0)
+ {
+ node->as_whichplan = n;
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
}
}
@@ -277,6 +337,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 9121537..e4f2bb6 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1170,6 +1170,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
1.8.3.1
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: Text/Plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline; filename="gentbl.sql"
CREATE EXTENSION postgres_fdw;
CREATE TABLE pl (a int, b int);
CREATE TABLE cl1 (LIKE pl) INHERITS (pl);
CREATE TABLE cl2 (LIKE pl) INHERITS (pl);
CREATE TABLE cl3 (LIKE pl) INHERITS (pl);
CREATE TABLE cl4 (LIKE pl) INHERITS (pl);
INSERT INTO cl1 (SELECT a, a FROM generate_series(0000000, 0999999) a);
INSERT INTO cl2 (SELECT a, a FROM generate_series(1000000, 1999999) a);
INSERT INTO cl3 (SELECT a, a FROM generate_series(2000000, 2999999) a);
INSERT INTO cl4 (SELECT a, a FROM generate_series(3000000, 3999999) a);
CREATE TABLE t0 (LIKE pl);
INSERT INTO t0 (SELECT a, a FROM generate_series(0000000, 9999999) a);
CREATE SERVER sv0 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv1 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv2 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv3 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv4 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE USER MAPPING FOR public SERVER sv0;
CREATE USER MAPPING FOR public SERVER sv1;
CREATE USER MAPPING FOR public SERVER sv2;
CREATE USER MAPPING FOR public SERVER sv3;
CREATE USER MAPPING FOR public SERVER sv4;
CREATE FOREIGN TABLE ft10 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl1');
CREATE FOREIGN TABLE ft20 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl2');
CREATE FOREIGN TABLE ft30 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl3');
CREATE FOREIGN TABLE ft40 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl4');
CREATE FOREIGN TABLE ft11 (a int, b int) SERVER sv1 OPTIONS (table_name 'cl1');
CREATE FOREIGN TABLE ft22 (a int, b int) SERVER sv2 OPTIONS (table_name 'cl2');
CREATE FOREIGN TABLE ft33 (a int, b int) SERVER sv3 OPTIONS (table_name 'cl3');
CREATE FOREIGN TABLE ft44 (a int, b int) SERVER sv4 OPTIONS (table_name 'cl4');
CREATE TABLE pf0 (LIKE pl);
ALTER FOREIGN TABLE ft10 INHERIT pf0;
ALTER FOREIGN TABLE ft20 INHERIT pf0;
ALTER FOREIGN TABLE ft30 INHERIT pf0;
ALTER FOREIGN TABLE ft40 INHERIT pf0;
CREATE table pf1 (LIKE pl);
ALTER FOREIGN TABLE ft11 INHERIT pf1;
ALTER FOREIGN TABLE ft22 INHERIT pf1;
ALTER FOREIGN TABLE ft33 INHERIT pf1;
ALTER FOREIGN TABLE ft44 INHERIT pf1;
ANALYZE;
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: Text/Plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline; filename="testrun.sh"
#! /bin/bash
function do_test() {
echo $1
for i in $(seq 1 10); do psql postgres -c "set log_min_duration_statement = 0; set client_min_messages=log; select sum(a) from $1"; done | grep LOG
}
for i in "t0" "pl" "pf0" "pf1";
do
do_test $i
done
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Wed_Jul_06_16_29_14_2016_599)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 67 +++++++++++++++++++++++++++++++++++++--
src/include/nodes/execnodes.h | 2 ++
2 files changed, 67 insertions(+), 2 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..9a4063a 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,6 +199,8 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
+ int stopplan = node->as_whichplan;
+
for (;;)
{
PlanState *subnode;
@@ -207,7 +214,36 @@ ExecAppend(AppendState *node)
/*
* get a tuple from the subplan
*/
- result = ExecProcNode(subnode);
+ Assert(!node->as_finished[node->as_whichplan]);
+
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ /* Move to the next living node */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ /* No node is ready yet, return as not-ready */
+ if (node->as_whichplan == stopplan)
+ return;
+
+ /* Try the next node */
+ continue;
+ }
+
+ /* If not async, immediately wait for this subnode */
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ result = ExecConsumeResult((PlanState *) subnode);
if (!TupIsNull(result))
{
@@ -220,6 +256,31 @@ ExecAppend(AppendState *node)
return;
}
+ if (node->as_async)
+ {
+ node->as_finished[node->as_whichplan] = true;
+ stopplan = node->as_whichplan;
+
+ /* Find the next living subnode */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ if (node->as_whichplan != stopplan)
+ {
+ stopplan = node->as_whichplan;
+ continue;
+ }
+
+ /* All subnodes are exhausted. Finish this node. */
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
+ return;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
@@ -277,6 +338,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index b72decc..b0a86c5 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1177,6 +1177,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
2.9.2
----Next_Part(Mon_Aug_29_17_08_36_2016_213)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Mon_Aug_29_17_08_36_2016_213)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 94 ++++++++++++++++++++++++++++++++-------
src/include/nodes/execnodes.h | 2 +
2 files changed, 80 insertions(+), 16 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..1c0d26e 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,49 +199,104 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
- for (;;)
+ int n_notready = 0;
+ PlanState *subnode;
+ int i, n;
+
+ n = node->as_whichplan;
+
+ for (i = 0 ; i < node->as_nplans ; i++)
{
- PlanState *subnode;
TupleTableSlot *result;
+ if (node->as_async)
+ {
+ if (n >= node->as_nplans)
+ n = 0;
+
+ if (node->as_finished[n])
+ {
+ n++;
+ continue;
+ }
+ }
+
/*
* figure out which subplan we are currently processing
*/
- subnode = node->appendplans[node->as_whichplan];
+ subnode = node->appendplans[n];
/*
- * get a tuple from the subplan
+ * execute the subplan to get a result if it is not ready yet
*/
- result = ExecProcNode(subnode);
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ /*
+ * This subnode is not ready yet when asynchrony is not allowed,
+ * immediately wait for this subnode.
+ */
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ n_notready++;
+ n++;
+ continue;
+ }
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ Assert(subnode->result_ready);
+
+ result = ExecConsumeResult((PlanState *)subnode);
if (!TupIsNull(result))
{
/*
- * If the subplan gave us something then return it as-is. We do
- * NOT make use of the result slot that was set up in
- * ExecInitAppend; there's no need for it.
+ * If the subplan gave us something then return it
+ * as-is. We do NOT make use of the result slot that was
+ * set up in ExecInitAppend; there's no need for it.
*/
+ node->as_whichplan = n;
ExecReturnTuple(&node->ps, result);
return;
}
+ /* Tuple has been exhausted on this subnode */
+ if (node->as_async)
+ {
+ node->as_finished[n++] = true;
+ continue;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
* ExecInitAppend.
*/
if (ScanDirectionIsForward(node->ps.state->es_direction))
- node->as_whichplan++;
+ {
+ if (n >= node->as_nplans - 1)
+ break;
+ n++;
+ }
else
- node->as_whichplan--;
- if (!exec_append_initialize_next(node))
{
- ExecReturnTuple(&node->ps,
- ExecClearTuple(node->ps.ps_ResultTupleSlot));
- return;
+ if (n == 0)
+ break;
+ n--;
}
+ }
- /* Else loop back and try to get a tuple from the new subplan */
+ /*
+ * We are finished if reached here and no subnodes are not-ready
+ */
+ if (n_notready == 0)
+ {
+ node->as_whichplan = n;
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
}
}
@@ -277,6 +337,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 9121537..e4f2bb6 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1170,6 +1170,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
1.8.3.1
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: Text/Plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline; filename="gentbl.sql"
CREATE EXTENSION postgres_fdw;
CREATE TABLE pl (a int, b int);
CREATE TABLE cl1 (LIKE pl) INHERITS (pl);
CREATE TABLE cl2 (LIKE pl) INHERITS (pl);
CREATE TABLE cl3 (LIKE pl) INHERITS (pl);
CREATE TABLE cl4 (LIKE pl) INHERITS (pl);
INSERT INTO cl1 (SELECT a, a FROM generate_series(0000000, 0999999) a);
INSERT INTO cl2 (SELECT a, a FROM generate_series(1000000, 1999999) a);
INSERT INTO cl3 (SELECT a, a FROM generate_series(2000000, 2999999) a);
INSERT INTO cl4 (SELECT a, a FROM generate_series(3000000, 3999999) a);
CREATE TABLE t0 (LIKE pl);
INSERT INTO t0 (SELECT a, a FROM generate_series(0000000, 9999999) a);
CREATE SERVER sv0 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv1 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv2 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv3 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv4 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE USER MAPPING FOR public SERVER sv0;
CREATE USER MAPPING FOR public SERVER sv1;
CREATE USER MAPPING FOR public SERVER sv2;
CREATE USER MAPPING FOR public SERVER sv3;
CREATE USER MAPPING FOR public SERVER sv4;
CREATE FOREIGN TABLE ft10 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl1');
CREATE FOREIGN TABLE ft20 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl2');
CREATE FOREIGN TABLE ft30 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl3');
CREATE FOREIGN TABLE ft40 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl4');
CREATE FOREIGN TABLE ft11 (a int, b int) SERVER sv1 OPTIONS (table_name 'cl1');
CREATE FOREIGN TABLE ft22 (a int, b int) SERVER sv2 OPTIONS (table_name 'cl2');
CREATE FOREIGN TABLE ft33 (a int, b int) SERVER sv3 OPTIONS (table_name 'cl3');
CREATE FOREIGN TABLE ft44 (a int, b int) SERVER sv4 OPTIONS (table_name 'cl4');
CREATE TABLE pf0 (LIKE pl);
ALTER FOREIGN TABLE ft10 INHERIT pf0;
ALTER FOREIGN TABLE ft20 INHERIT pf0;
ALTER FOREIGN TABLE ft30 INHERIT pf0;
ALTER FOREIGN TABLE ft40 INHERIT pf0;
CREATE table pf1 (LIKE pl);
ALTER FOREIGN TABLE ft11 INHERIT pf1;
ALTER FOREIGN TABLE ft22 INHERIT pf1;
ALTER FOREIGN TABLE ft33 INHERIT pf1;
ALTER FOREIGN TABLE ft44 INHERIT pf1;
ANALYZE;
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: Text/Plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline; filename="testrun.sh"
#! /bin/bash
function do_test() {
echo $1
for i in $(seq 1 10); do psql postgres -c "set log_min_duration_statement = 0; set client_min_messages=log; select sum(a) from $1"; done | grep LOG
}
for i in "t0" "pl" "pf0" "pf1";
do
do_test $i
done
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Wed_Jul_06_16_29_14_2016_599)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 67 +++++++++++++++++++++++++++++++++++++--
src/include/nodes/execnodes.h | 2 ++
2 files changed, 67 insertions(+), 2 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..9a4063a 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,6 +199,8 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
+ int stopplan = node->as_whichplan;
+
for (;;)
{
PlanState *subnode;
@@ -207,7 +214,36 @@ ExecAppend(AppendState *node)
/*
* get a tuple from the subplan
*/
- result = ExecProcNode(subnode);
+ Assert(!node->as_finished[node->as_whichplan]);
+
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ /* Move to the next living node */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ /* No node is ready yet, return as not-ready */
+ if (node->as_whichplan == stopplan)
+ return;
+
+ /* Try the next node */
+ continue;
+ }
+
+ /* If not async, immediately wait for this subnode */
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ result = ExecConsumeResult((PlanState *) subnode);
if (!TupIsNull(result))
{
@@ -220,6 +256,31 @@ ExecAppend(AppendState *node)
return;
}
+ if (node->as_async)
+ {
+ node->as_finished[node->as_whichplan] = true;
+ stopplan = node->as_whichplan;
+
+ /* Find the next living subnode */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ if (node->as_whichplan != stopplan)
+ {
+ stopplan = node->as_whichplan;
+ continue;
+ }
+
+ /* All subnodes are exhausted. Finish this node. */
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
+ return;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
@@ -277,6 +338,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index b72decc..b0a86c5 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1177,6 +1177,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
1.8.3.1
----Next_Part(Thu_Jul_21_18_50_07_2016_597)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Thu_Jul_21_18_50_07_2016_597)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 67 +++++++++++++++++++++++++++++++++++++--
src/include/nodes/execnodes.h | 2 ++
2 files changed, 67 insertions(+), 2 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..9a4063a 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,6 +199,8 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
+ int stopplan = node->as_whichplan;
+
for (;;)
{
PlanState *subnode;
@@ -207,7 +214,36 @@ ExecAppend(AppendState *node)
/*
* get a tuple from the subplan
*/
- result = ExecProcNode(subnode);
+ Assert(!node->as_finished[node->as_whichplan]);
+
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ /* Move to the next living node */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ /* No node is ready yet, return as not-ready */
+ if (node->as_whichplan == stopplan)
+ return;
+
+ /* Try the next node */
+ continue;
+ }
+
+ /* If not async, immediately wait for this subnode */
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ result = ExecConsumeResult((PlanState *) subnode);
if (!TupIsNull(result))
{
@@ -220,6 +256,31 @@ ExecAppend(AppendState *node)
return;
}
+ if (node->as_async)
+ {
+ node->as_finished[node->as_whichplan] = true;
+ stopplan = node->as_whichplan;
+
+ /* Find the next living subnode */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ if (node->as_whichplan != stopplan)
+ {
+ stopplan = node->as_whichplan;
+ continue;
+ }
+
+ /* All subnodes are exhausted. Finish this node. */
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
+ return;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
@@ -277,6 +338,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index b72decc..b0a86c5 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1177,6 +1177,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
1.8.3.1
----Next_Part(Thu_Jul_21_18_50_07_2016_597)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Thu_Jul_21_18_50_07_2016_597)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 67 +++++++++++++++++++++++++++++++++++++--
src/include/nodes/execnodes.h | 2 ++
2 files changed, 67 insertions(+), 2 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..9a4063a 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,6 +199,8 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
+ int stopplan = node->as_whichplan;
+
for (;;)
{
PlanState *subnode;
@@ -207,7 +214,36 @@ ExecAppend(AppendState *node)
/*
* get a tuple from the subplan
*/
- result = ExecProcNode(subnode);
+ Assert(!node->as_finished[node->as_whichplan]);
+
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ /* Move to the next living node */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ /* No node is ready yet, return as not-ready */
+ if (node->as_whichplan == stopplan)
+ return;
+
+ /* Try the next node */
+ continue;
+ }
+
+ /* If not async, immediately wait for this subnode */
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ result = ExecConsumeResult((PlanState *) subnode);
if (!TupIsNull(result))
{
@@ -220,6 +256,31 @@ ExecAppend(AppendState *node)
return;
}
+ if (node->as_async)
+ {
+ node->as_finished[node->as_whichplan] = true;
+ stopplan = node->as_whichplan;
+
+ /* Find the next living subnode */
+ do
+ {
+ node->as_whichplan =
+ (node->as_whichplan + 1) % node->as_nplans;
+ } while (node->as_whichplan != stopplan &&
+ node->as_finished[node->as_whichplan]);
+
+ if (node->as_whichplan != stopplan)
+ {
+ stopplan = node->as_whichplan;
+ continue;
+ }
+
+ /* All subnodes are exhausted. Finish this node. */
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
+ return;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
@@ -277,6 +338,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index b72decc..b0a86c5 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1177,6 +1177,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
1.8.3.1
----Next_Part(Thu_Jul_21_18_50_07_2016_597)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Thu_Jul_21_18_50_07_2016_597)----
^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 7/7] Make Append node async-aware.
@ 2016-06-28 09:52 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 67+ messages in thread
From: Kyotaro Horiguchi @ 2016-06-28 09:52 UTC (permalink / raw)
Change append node to be capable to handle asynchronous children
properly. As soon as it receives !async_ready from a child, it moves
to the next child and if no child is ready, it sleeps until at least
one of them become ready.
---
src/backend/executor/nodeAppend.c | 94 ++++++++++++++++++++++++++++++++-------
src/include/nodes/execnodes.h | 2 +
2 files changed, 80 insertions(+), 16 deletions(-)
diff --git a/src/backend/executor/nodeAppend.c b/src/backend/executor/nodeAppend.c
index e0ce8c6..1c0d26e 100644
--- a/src/backend/executor/nodeAppend.c
+++ b/src/backend/executor/nodeAppend.c
@@ -58,6 +58,7 @@
#include "postgres.h"
#include "executor/execdebug.h"
+#include "executor/execAsync.h"
#include "executor/nodeAppend.h"
static bool exec_append_initialize_next(AppendState *appendstate);
@@ -121,6 +122,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
AppendState *appendstate = makeNode(AppendState);
PlanState **appendplanstates;
+ bool *finished;
int nplans;
int i;
ListCell *lc;
@@ -134,14 +136,17 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
nplans = list_length(node->appendplans);
appendplanstates = (PlanState **) palloc0(nplans * sizeof(PlanState *));
-
+ finished = (bool *) palloc0(nplans * sizeof(bool));
+
/*
* create new AppendState for our append node
*/
appendstate->ps.plan = (Plan *) node;
appendstate->ps.state = estate;
appendstate->appendplans = appendplanstates;
+ appendstate->as_finished = finished;
appendstate->as_nplans = nplans;
+ appendstate->as_async = ((eflags & EXEC_FLAG_BACKWARD) == 0);
/*
* Miscellaneous initialization
@@ -194,49 +199,104 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
void
ExecAppend(AppendState *node)
{
- for (;;)
+ int n_notready = 0;
+ PlanState *subnode;
+ int i, n;
+
+ n = node->as_whichplan;
+
+ for (i = 0 ; i < node->as_nplans ; i++)
{
- PlanState *subnode;
TupleTableSlot *result;
+ if (node->as_async)
+ {
+ if (n >= node->as_nplans)
+ n = 0;
+
+ if (node->as_finished[n])
+ {
+ n++;
+ continue;
+ }
+ }
+
/*
* figure out which subplan we are currently processing
*/
- subnode = node->appendplans[node->as_whichplan];
+ subnode = node->appendplans[n];
/*
- * get a tuple from the subplan
+ * execute the subplan to get a result if it is not ready yet
*/
- result = ExecProcNode(subnode);
+ if (!subnode->result_ready)
+ ExecExecuteNode(subnode);
+
+ /*
+ * This subnode is not ready yet when asynchrony is not allowed,
+ * immediately wait for this subnode.
+ */
+ if (!subnode->result_ready)
+ {
+ if (node->as_async)
+ {
+ n_notready++;
+ n++;
+ continue;
+ }
+ ExecAsyncWaitForNode(subnode);
+ }
+
+ Assert(subnode->result_ready);
+
+ result = ExecConsumeResult((PlanState *)subnode);
if (!TupIsNull(result))
{
/*
- * If the subplan gave us something then return it as-is. We do
- * NOT make use of the result slot that was set up in
- * ExecInitAppend; there's no need for it.
+ * If the subplan gave us something then return it
+ * as-is. We do NOT make use of the result slot that was
+ * set up in ExecInitAppend; there's no need for it.
*/
+ node->as_whichplan = n;
ExecReturnTuple(&node->ps, result);
return;
}
+ /* Tuple has been exhausted on this subnode */
+ if (node->as_async)
+ {
+ node->as_finished[n++] = true;
+ continue;
+ }
+
/*
* Go on to the "next" subplan in the appropriate direction. If no
* more subplans, return the empty slot set up for us by
* ExecInitAppend.
*/
if (ScanDirectionIsForward(node->ps.state->es_direction))
- node->as_whichplan++;
+ {
+ if (n >= node->as_nplans - 1)
+ break;
+ n++;
+ }
else
- node->as_whichplan--;
- if (!exec_append_initialize_next(node))
{
- ExecReturnTuple(&node->ps,
- ExecClearTuple(node->ps.ps_ResultTupleSlot));
- return;
+ if (n == 0)
+ break;
+ n--;
}
+ }
- /* Else loop back and try to get a tuple from the new subplan */
+ /*
+ * We are finished if reached here and no subnodes are not-ready
+ */
+ if (n_notready == 0)
+ {
+ node->as_whichplan = n;
+ ExecReturnTuple(&node->ps,
+ ExecClearTuple(node->ps.ps_ResultTupleSlot));
}
}
@@ -277,6 +337,8 @@ ExecReScanAppend(AppendState *node)
{
PlanState *subnode = node->appendplans[i];
+ node->as_finished[i] = false;
+
/*
* ExecReScan doesn't know about my subplans, so I have to do
* changed-parameter signaling myself.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 9121537..e4f2bb6 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1170,6 +1170,8 @@ typedef struct AppendState
{
PlanState ps; /* its first field is NodeTag */
PlanState **appendplans; /* array of PlanStates for my inputs */
+ bool as_async; /* true to allow async execution */
+ bool *as_finished; /* array of the running state of subplans */
int as_nplans;
int as_whichplan;
} AppendState;
--
1.8.3.1
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: Text/Plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline; filename="gentbl.sql"
CREATE EXTENSION postgres_fdw;
CREATE TABLE pl (a int, b int);
CREATE TABLE cl1 (LIKE pl) INHERITS (pl);
CREATE TABLE cl2 (LIKE pl) INHERITS (pl);
CREATE TABLE cl3 (LIKE pl) INHERITS (pl);
CREATE TABLE cl4 (LIKE pl) INHERITS (pl);
INSERT INTO cl1 (SELECT a, a FROM generate_series(0000000, 0999999) a);
INSERT INTO cl2 (SELECT a, a FROM generate_series(1000000, 1999999) a);
INSERT INTO cl3 (SELECT a, a FROM generate_series(2000000, 2999999) a);
INSERT INTO cl4 (SELECT a, a FROM generate_series(3000000, 3999999) a);
CREATE TABLE t0 (LIKE pl);
INSERT INTO t0 (SELECT a, a FROM generate_series(0000000, 9999999) a);
CREATE SERVER sv0 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv1 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv2 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv3 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE SERVER sv4 FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '/tmp', dbname 'postgres');
CREATE USER MAPPING FOR public SERVER sv0;
CREATE USER MAPPING FOR public SERVER sv1;
CREATE USER MAPPING FOR public SERVER sv2;
CREATE USER MAPPING FOR public SERVER sv3;
CREATE USER MAPPING FOR public SERVER sv4;
CREATE FOREIGN TABLE ft10 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl1');
CREATE FOREIGN TABLE ft20 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl2');
CREATE FOREIGN TABLE ft30 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl3');
CREATE FOREIGN TABLE ft40 (a int, b int) SERVER sv0 OPTIONS (table_name 'cl4');
CREATE FOREIGN TABLE ft11 (a int, b int) SERVER sv1 OPTIONS (table_name 'cl1');
CREATE FOREIGN TABLE ft22 (a int, b int) SERVER sv2 OPTIONS (table_name 'cl2');
CREATE FOREIGN TABLE ft33 (a int, b int) SERVER sv3 OPTIONS (table_name 'cl3');
CREATE FOREIGN TABLE ft44 (a int, b int) SERVER sv4 OPTIONS (table_name 'cl4');
CREATE TABLE pf0 (LIKE pl);
ALTER FOREIGN TABLE ft10 INHERIT pf0;
ALTER FOREIGN TABLE ft20 INHERIT pf0;
ALTER FOREIGN TABLE ft30 INHERIT pf0;
ALTER FOREIGN TABLE ft40 INHERIT pf0;
CREATE table pf1 (LIKE pl);
ALTER FOREIGN TABLE ft11 INHERIT pf1;
ALTER FOREIGN TABLE ft22 INHERIT pf1;
ALTER FOREIGN TABLE ft33 INHERIT pf1;
ALTER FOREIGN TABLE ft44 INHERIT pf1;
ANALYZE;
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: Text/Plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline; filename="testrun.sh"
#! /bin/bash
function do_test() {
echo $1
for i in $(seq 1 10); do psql postgres -c "set log_min_duration_statement = 0; set client_min_messages=log; select sum(a) from $1"; done | grep LOG
}
for i in "t0" "pl" "pf0" "pf1";
do
do_test $i
done
----Next_Part(Wed_Jul_06_16_29_14_2016_599)--
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
----Next_Part(Wed_Jul_06_16_29_14_2016_599)----
^ permalink raw reply [nested|flat] 67+ messages in thread
end of thread, other threads:[~2016-06-28 09:52 UTC | newest]
Thread overview: 67+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2005-01-30 06:14 Huge memory consumption during vacuum (v.8.0) Oleg Bartunov <[email protected]>
2005-01-30 06:51 ` Tom Lane <[email protected]>
2005-01-30 06:58 ` Oleg Bartunov <[email protected]>
2005-01-30 07:04 ` Tom Lane <[email protected]>
2005-01-30 07:10 ` Oleg Bartunov <[email protected]>
2005-01-30 07:23 ` Oleg Bartunov <[email protected]>
2005-02-01 20:11 ` Oleg Bartunov <[email protected]>
2005-02-01 21:26 ` Tom Lane <[email protected]>
2005-02-01 21:30 ` Oleg Bartunov <[email protected]>
2005-02-02 07:27 ` Oleg Bartunov <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
2016-06-28 09:52 [PATCH 7/7] Make Append node async-aware. Kyotaro Horiguchi <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox