Subject: Re: asynchronous and vectorized execution
To: Robert Haas <robertmhaas@gmail.com>
References: 
 <CA+Tgmobx8su_bYtAa3DgrqB+R7xZG6kHRj0ccMUUshKAQVftww@mail.gmail.com>
 <49D1AF71-15BD-42C2-88FD-5E894EE4C5B8@postgrespro.ru>
 <CA+TgmoZm+kHNYtFh7DBz8Dn3Rxxxr79OAagYtJSbj5_SMh4wAQ@mail.gmail.com>
 <5732399A.4050203@postgrespro.ru>
 <CA+TgmoYcmXWCvjtP6Ou1ChYKK2CvjG9gw-f7tbckaJ7KDhwHng@mail.gmail.com>
Cc: "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
From: Konstantin Knizhnik <k.knizhnik@postgrespro.ru>
Message-ID: <57333EFE.3@postgrespro.ru>
Date: Wed, 11 May 2016 17:17:34 +0300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
 Thunderbird/38.3.0
MIME-Version: 1.0
In-Reply-To: 
 <CA+TgmoYcmXWCvjtP6Ou1ChYKK2CvjG9gw-f7tbckaJ7KDhwHng@mail.gmail.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org


On 11.05.2016 17:00, Robert Haas wrote:
> On Tue, May 10, 2016 at 3:42 PM, Konstantin Knizhnik
> <k.knizhnik@postgrespro.ru> wrote:
>> Doesn't this actually mean that we need to have normal job scheduler which
>> is given queue of jobs and having some pool of threads will be able to
>> orginize efficient execution of queries? Optimizer can build pipeline
>> (graph) of tasks, which corresponds to execution plan nodes, i.e. SeqScan,
>> Sort, ... Each task is splitted into several jobs which can be concurretly
>> scheduled by task dispatcher.  So you will not have blocked worker waiting
>> for something and all system resources will be utilized. Such approach with
>> dispatcher allows to implement quotas, priorities,... Also dispatches can
>> care about NUMA and cache optimizations which is especially critical on
>> modern architectures. One more reference:
>> http://db.in.tum.de/~leis/papers/morsels.pdf
> I read this as a proposal to redesign the entire optimizer and
> executor to use some new kind of plan.  That's not a project I'm
> willing to entertain; it is hard to imagine we could do it in a
> reasonable period of time without introducing bugs and performance
> regressions.  I think there is a great deal of performance benefit
> that we can get by changing things incrementally.
>
Yes, I agree with you that complete rewriting of optimizer is huge 
project with unpredictable influence on performance of some queries.
Changing things incrementally is good approach, but only if we are 
moving in right direction.
I still not sure that introduction of async. operations is step in right 
direction. Async.ops are used to significantly complicate code (since 
you have to maintain state yourself). It will be bad if implementation 
of each node has to deal with async state itself in its own manner.

My suggestion is to try to provide some generic mechanism for managing 
state transition and have some scheduler which controls this process. It 
should not be responsibility of node implementation to organize 
asynchronous/parallel execution. Instead of this it should just produce 
set of jobs which execution should  be controlled by scheduler. First 
implementation of scheduler can be quite simple. But later in can become 
more clever: try to bind data to processors and do many other 
optimizations.


-- 
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers