From: Tomas Vondra <tomas.vondra@postgresql.org>
To: pgsql-committers@lists.postgresql.org
Subject: pgsql: Allocate hash join files in a separate memory context
Date: Fri, 19 May 2023 15:18:40 +0000
Message-ID: <E1q01sO-000hAl-JI@gemulon.postgresql.org> (raw)
Allocate hash join files in a separate memory context
Should a hash join exceed memory limit, the hashtable is split up into
multiple batches. The number of batches is doubled each time a given
batch is determined not to fit in memory. Each batch file is allocated
with a block-sized buffer for buffering tuples and parallel hash join
has additional sharedtuplestore accessor buffers.
In some pathological cases requiring a lot of batches, often with skewed
data, bad stats, or very large datasets, users can run out-of-memory
solely from the memory overhead of all the batch files' buffers.
Batch files were allocated in the ExecutorState memory context, making
it very hard to identify when this batch explosion was the source of an
OOM. This commit allocates the batch files in a dedicated memory
context, making it easier to identify the cause of an OOM and work to
avoid it.
Based on initial draft by Tomas Vondra, with significant reworks and
improvements by Jehan-Guillaume de Rorthais.
Author: Jehan-Guillaume de Rorthais <jgdr@dalibo.com>
Author: Tomas Vondra <tomas.vondra@enterprisedb.com>
Reviewed-by: Melanie Plageman <melanieplageman@gmail.com>
Discussion: https://postgr.es/m/20190421114618.z3mpgmimc3rmubi4@development
Discussion: https://postgr.es/m/20230504193006.1b5b9622%40karst#273020ff4061fc7a2fbb1ba96b281f17
Branch
------
master
Details
-------
https://git.postgresql.org/pg/commitdiff/8c4040edf456d9241816176eacb79e4d9a0034fc
Modified Files
--------------
src/backend/executor/nodeHash.c | 46 ++++++++++++++++++++++---------
src/backend/executor/nodeHashjoin.c | 31 +++++++++++++++++----
src/backend/utils/sort/sharedtuplestore.c | 8 ++++++
src/include/executor/hashjoin.h | 30 ++++++++++++++------
src/include/executor/nodeHashjoin.h | 2 +-
5 files changed, 88 insertions(+), 29 deletions(-)
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-committers@postgresql.org
Cc: tomas.vondra@postgresql.org, pgsql-committers@lists.postgresql.org
Subject: Re: pgsql: Allocate hash join files in a separate memory context
In-Reply-To: <E1q01sO-000hAl-JI@gemulon.postgresql.org>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox