From: Tomas Vondra <tomas.vondra@2ndquadrant.com>
To: Melanie Plageman <melanieplageman@gmail.com>
Cc: Thomas Munro <thomas.munro@gmail.com>
Cc: hackers <pgsql-hackers@postgresql.org>
Subject: Re: accounting for memory used for BufFile during hash joins
Date: Wed, 8 May 2019 15:34:38 +0200
Message-ID: <20190508133438.7nfzyioihjcvkjwi@development> (raw)
In-Reply-To: <CAAKRu_Z-=Ahu-M-8FHmSCMpu4wumRN0dfvDrUoJ0s6__5ZYcQg@mail.gmail.com>
References: <20190504003414.bulcbnge3rhwhcsh@development>
<CAAKRu_ZTy=a2jqj0++jDwke1_ssJTBB0ezU2t8OwogmXDbYdYw@mail.gmail.com>
<CA+hUKG+ppih9DtZwGJh4KMZXdtD=c9+fYoS1CSeBQBs9JzZtbA@mail.gmail.com>
<20190507031511.lvp6bqn2e5dpsnga@development>
<CA+hUKG+546WK=ya70QMs-oWE94BhsUKvqjr=ds_X_wkv7fP-2w@mail.gmail.com>
<20190507135912.qakpj74sv3ngrypz@development>
<CAAKRu_Z-=Ahu-M-8FHmSCMpu4wumRN0dfvDrUoJ0s6__5ZYcQg@mail.gmail.com>
On Tue, May 07, 2019 at 05:43:56PM -0700, Melanie Plageman wrote:
> On Tue, May 7, 2019 at 6:59 AM Tomas Vondra <tomas.vondra@2ndquadrant.com>
> wrote:
>
> On Tue, May 07, 2019 at 04:28:36PM +1200, Thomas Munro wrote:
> >On Tue, May 7, 2019 at 3:15 PM Tomas Vondra
> ><tomas.vondra@2ndquadrant.com> wrote:
> >> On Tue, May 07, 2019 at 01:48:40PM +1200, Thomas Munro wrote:
> >> Switching to some other algorithm during execution moves the goal
> posts
> >> to the next galaxy, I'm afraid.
> >
> >The main problem I'm aware of with sort-merge join is: not all that is
> >hashable is sortable. So BNL is actually the only solution I'm aware
> >of for problem B that doesn't involve changing a fundamental thing
> >about PostgreSQL's data type requirements.
> >
>
> Sure, each of those algorithms has limitations. But I think that's
> mostly
> irrelevant to the main issue - switching between algorithms
> mid-execution.
> At that point some of the tuples might have been already sent sent to
> the
> other nodes, and I have no idea how to "resume" the tuple stream short
> of
> buffering everything locally until the join completes. And that would be
> rather terrible, I guess.
>
> What if you switched to NLJ on a batch-by-batch basis and did it before
> starting
> execution of the join but after building the inner side of the hash
> table. That
> way, no tuples will have been sent to other nodes yet.
>
Interesting idea! I think you're right doing it on a per-batch basis
would solve that problem. Essentially, if all (or >95%) of the tuples
has the same hash value, we could switch to a special "degraded" mode
doing something like a NL. At that point the hash table benefits are
lost anyway, because all the tuples are in a single chain, so it's not
going to be much slower.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: tomas.vondra@2ndquadrant.com, melanieplageman@gmail.com, thomas.munro@gmail.com
Subject: Re: accounting for memory used for BufFile during hash joins
In-Reply-To: <20190508133438.7nfzyioihjcvkjwi@development>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox