Message-ID: <3d8ffaa5-b9a1-9538-9ac3-ffa751449f4b@enterprisedb.com>
Date: Thu, 8 Jun 2023 12:37:37 +0200
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.10.0
Subject: Re: Let's make PostgreSQL multi-threaded
Content-Language: en-US
To: Thomas Munro <thomas.munro@gmail.com>,
 Jeremy Schneider <schneider@ardentperf.com>
Cc: pgsql-hackers@lists.postgresql.org, Thomas Kellerer <shammat@gmx.net>,
 Heikki Linnakangas <hlinnaka@iki.fi>, Tom Lane <tgl@sss.pgh.pa.us>
References: <31cc6df9-53fe-3cd9-af5b-ac0d801163f4@iki.fi>
 <4178104.1685978307@sss.pgh.pa.us>
 <4ce6c0f8-e8a4-1672-93fd-49d3fa975ee5@iki.fi>
 <29fe5f48-a6ed-d896-45ed-16b5904353a9@enterprisedb.com>
 <41c1e20d-f179-f87e-5929-80ca9ee0c105@gmx.net>
 <d9656e32-f1ea-5401-d07b-dab34569236a@ardentperf.com>
 <CA+hUKGLuV68xKe9JwFkeQJc9A2vSe8+nhahAF1my4Qg=UD2PwQ@mail.gmail.com>
From: Tomas Vondra <tomas.vondra@enterprisedb.com>
In-Reply-To: <CA+hUKGLuV68xKe9JwFkeQJc9A2vSe8+nhahAF1my4Qg=UD2PwQ@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Archived-At: <https://www.postgresql.org/message-id/3d8ffaa5-b9a1-9538-9ac3-ffa751449f4b%40enterprisedb.com>
Precedence: bulk


On 6/8/23 01:37, Thomas Munro wrote:
> On Thu, Jun 8, 2023 at 10:37 AM Jeremy Schneider
> <schneider@ardentperf.com> wrote:
>> On 6/7/23 2:39 PM, Thomas Kellerer wrote:
>>> Tomas Vondra schrieb am 07.06.2023 um 21:20:
>>>> Also, which other projects did this transition? Is there something we
>>>> could learn from them? Were they restricted to much smaller list of
>>>> platforms?
>>>
>>> Not open source, but Oracle was historically multi-threaded on Windows
>>> and multi-process on all other platforms.
>>> I _think_ starting with 19c you can optionally run it multi-threaded on
>>> Linux as well.
>> Looks like it actually became publicly available in 12c. AFAICT Oracle
>> supports both modes today, with a config parameter to switch between them.
> 
> It's old, but this describes the 4 main models and which well known
> RDBMSes use them in section 2.3:
> 
> https://dsf.berkeley.edu/papers/fntdb07-architecture.pdf
> 
> TL;DR DB2 is the winner, it can do process-per-connection,
> thread-per-connection, process-pool or thread-pool.
> 

I think the basic architectures are known, especially from the user
perspective. I'm more interested in challenges the projects faced while
moving from one architecture to the other, or how / why they support
more than just one, etc.

In [1] Heikki argued that:

    I don't think this is worth it, unless we plan to eventually remove
    the multi-process mode. ... As long as you need to also support
    processes, you need to code to the lowest common denominator and
    don't really get the benefits.

But these projects clearly support multiple architectures, and have no
intention to ditch some of them. So how did they do that? Surely they
think there are benefits.

One option would be to just have separate code paths for processes and
threads, but the effort required to maintain and improve that would be
deadly. So the only feasible option seems to be they managed to abstract
the subsystems enough for the "regular" code to not care about model.


[1]
https://www.postgresql.org/message-id/6e3082dc-ff29-9cbf-847e-5f570828b46b@iki.fi

> I understand this thread to be about thread-per-connection (= backend,
> session, socket) for now.

Maybe, although people also proposed to switch the parallel query to
threads (so that'd be multiple threads per session). But I don't think
it really matters, the concerns are mostly about moving from one
architecture to another and/or supporting both.

regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company