Date: Fri, 14 Apr 2023 13:43:21 +0200
From: Jehan-Guillaume de Rorthais <jgdr@dalibo.com>
To: Matthias van de Meent <boekewurm+postgres@gmail.com>
Cc: Konstantin Knizhnik <knizhnik@garret.ru>, PostgreSQL Hackers
 <pgsql-hackers@postgresql.org>
Subject: Re: OOM in hash join
Message-ID: <20230414134321.46fd18d3@karst>
In-Reply-To: <CAEze2Wi1F3GuPapyXe54k29JqzbzQ5Mc3mng=nFSOYyonQ+NTw@mail.gmail.com>
References: <94608e6b-caca-02d2-1bec-9806532c476d@garret.ru>
	<CAEze2Wi1F3GuPapyXe54k29JqzbzQ5Mc3mng=nFSOYyonQ+NTw@mail.gmail.com>
Organization: Dalibo
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Archived-At: <https://www.postgresql.org/message-id/20230414134321.46fd18d3%40karst>
Precedence: bulk

On Fri, 14 Apr 2023 13:21:05 +0200
Matthias van de Meent <boekewurm+postgres@gmail.com> wrote:

> On Fri, 14 Apr 2023 at 12:59, Konstantin Knizhnik <knizhnik@garret.ru> wrote:
> >
> > Hi hackers,
> >
> > Too small value of work_mem cause memory overflow in parallel hash join
> > because of too much number batches.
> > There is the plan:  
> 
> [...]
> 
> > There is still some gap between size reported by memory context sump and
> > actual size of backend.
> > But is seems to be obvious, that trying to fit in work_mem
> > sharedtuplestore creates so much batches, that  them consume much more
> > memory than work_mem.

Indeed. The memory consumed by batches is not accounted and the consumption
reported in explain analyze is wrong.

Would you be able to test the latest patchset posted [1] ? This does not fix
the work_mem overflow, but it helps to keep the number of batches
balanced and acceptable. Any feedback, comment or review would be useful.

[1] https://www.postgresql.org/message-id/flat/20230408020119.32a0841b%40karst#616c1f41fcc10e8f89d41e8e5693618c

Regards,