From: Bruce Momjian <bruce@momjian.us>
To: Thomas Munro <thomas.munro@gmail.com>
Cc: pgsql-hackers <pgsql-hackers@postgresql.org>
Cc: Stephen Frost <sfrost@snowman.net>
Subject: Re: Append with naive multiplexing of FDWs
Date: Fri, 27 Sep 2019 12:20:02 -0400
Message-ID: <20190927162002.GC31412@momjian.us> (raw)
In-Reply-To: <CA+hUKGLBRyu0rHrDCMC4=Rn3252gogyp1SjOgG8SEKKZv=FwfQ@mail.gmail.com>
References: <CA+hUKGLBRyu0rHrDCMC4=Rn3252gogyp1SjOgG8SEKKZv=FwfQ@mail.gmail.com>
On Wed, Sep 4, 2019 at 06:18:31PM +1200, Thomas Munro wrote:
> Hello,
>
> A few years back[1] I experimented with a simple readiness API that
> would allow Append to start emitting tuples from whichever Foreign
> Scan has data available, when working with FDW-based sharding. I used
> that primarily as a way to test Andres's new WaitEventSet stuff and my
> kqueue implementation of that, but I didn't pursue it seriously
> because I knew we wanted a more ambitious async executor rewrite and
> many people had ideas about that, with schedulers capable of jumping
> all over the tree etc.
>
> Anyway, Stephen Frost pinged me off-list to ask about that patch, and
> asked why we don't just do this naive thing until we have something
> better. It's a very localised feature that works only between Append
> and its immediate children. The patch makes it work for postgres_fdw,
> but it should work for any FDW that can get its hands on a socket.
>
> Here's a quick rebase of that old POC patch, along with a demo. Since
> 2016, Parallel Append landed, but I didn't have time to think about
> how to integrate with that so I did a quick "sledgehammer" rebase that
> disables itself if parallelism is in the picture.
Yes, sharding has been waiting on parallel FDW scans. Would this work
for parallel partition scans if the partitions were FDWs?
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: bruce@momjian.us, thomas.munro@gmail.com, sfrost@snowman.net
Subject: Re: Append with naive multiplexing of FDWs
In-Reply-To: <20190927162002.GC31412@momjian.us>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox