Date: Tue, 5 Nov 2019 01:19:15 +0100
From: Tomas Vondra <tomas.vondra@2ndquadrant.com>
To: Gunther <raj@gusw.net>
Cc: "pgsql-performance@lists.postgresql.org"
 <pgsql-performance@lists.postgresql.org>
Subject: Re: FPGA optimization ...
Message-ID: <20191105001915.w7js3knyfvlznu6m@development>
References: <e41deea2-9594-36dc-be76-47c39368911e@gusw.net>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <e41deea2-9594-36dc-be76-47c39368911e@gusw.net>
User-Agent: NeoMutt/20180716
Precedence: bulk

On Mon, Nov 04, 2019 at 06:33:15PM -0500, Gunther wrote:
>The time has come.
>
>FPGA optimization is in the palm of our hands (literally a 2 TB 40 
>GB/s IO PostgreSQL server fits into less than a shoe box), and on 
>Amazon AWS F1 instances.
>
>Some demos are beginning to exist: https://github.com/Xilinx/data-analytics.
><https://github.com/Xilinx/data-analytics>
>
>But a lot more could be done. How about linear sort performance at 
>O(N)? https://hackaday.com/2016/01/20/a-linear-time-sorting-algorithm-for-fpgas/. 
>And how about ​https://people.csail.mit.edu/wjun/papers/fccm2017.pdf, 
>the following four sorting accelerators are used:
>
> * Tuple Sorter : Sorts an N-tuple using a sorting network.
> * Page Sorter : Sorts an 8KB (a flash page) chunk of sorted N-tuples
>   in on-chip memory.
> * Super-Page Sorter : Sorts 16 8K-32MB sorted chunks in DRAM.
> * Storage-to-Storage Sorter: Sorts 16 512MB or larger sorted chunks in
>   flash.
>
>Order of magnitude speed improvements? Better than Hadoop clusters on 
>a single chip? 40 GB/s I/O throughput massive full table scan, blazing 
>fast sort-merge joins? Here it is. Anybody working more on that? 
>Should be an ideal project for a student or a group of students.
>

For the record, this is not exactly a new thing. Netezza (a PostgreSQL
fork started in 1999 IBM) used FPGAs. Now there's swarm64 [1], another
PostgreSQL fork, also using FPGAs with newer PostgreSQL releases.

Those are proprietary forks, though. The main reason why the community
itself is not working on this directly (at least not on pgsql-hackers)
is exactly that it requires specialized hardware, which the devs
probably don't have, making development impossible, and the regular
customers are not asking for it either (one of the reasons being limited
availability of such hardware, especially for customers running in the
cloud and not being even able to deploy custom appliances).

I don't think this will change, unless the access to systems with FPGAs
becomes much easier (e.g. if AWS introduces such instance type).

>Is there a PostgreSQL foundation I could donate to, 501(c)(3) tax 
>exempt? I can donate and possibly find some people at Purdue 
>University who might take this on. Interest?
>

I don't think there's any such non-profit, managing/funding development.
At least I'm not avare of it. There are various non-profits around the
world, but those are organizing events and local communities.

I'd say the best way to do something like this is to either talk to one
of the companies participating in PostgreSQL devopment (pgsql-hackers is
probably a good starting point), or - if you absolutely need to go
through a non-profit - approach a university (which does not mean people
from pgsql-hackers can't be involved, of course). I've been involved in
a couple of such research projects in Europe, not sure what exactly is
the situation/rules in US.

regards

[1] https://swarm64.com/netezza-replacement/

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services