From: Andrei Lepikhov <lepihov@gmail.com>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Thomas Munro <thomas.munro@gmail.com>
Cc: Craig Milhiser <craig@milhiser.com>
Cc: pgsql-bugs@lists.postgresql.org
Subject: Re: Reference to - BUG #18349: ERROR: invalid DSA memory alloc request size 1811939328, CONTEXT: parallel worker
Date: Mon, 14 Oct 2024 16:16:11 +0700
Message-ID: <3a1ac6f1-ac30-4ed4-8d0d-beb8a5aca7e5@gmail.com> (raw)
In-Reply-To: <3581840.1728887169@sss.pgh.pa.us>
References: <CA+wnhO1OfgXbmXgC4fv_uu=OxcDQuHvfoQ4k0DFeB0Qqd-X-rQ@mail.gmail.com>
<CA+hUKG+YZpOyP-XXT65o9hx=oGF+9-BJzjqzRXcdvWV=ZrJbTA@mail.gmail.com>
<CA+wnhO2NDZFWB1TrZga-uLzpfSDjKH5Axa2d2+1Q76JSsS7yMg@mail.gmail.com>
<CA+wnhO0S2kxJ2_M57uJXRocMpcDtz6N4RkP-3Xy=pFs1wp7hmg@mail.gmail.com>
<CA+hUKGLPKWamfKjtEkPLvRDSeOqoZZ0wDfU65d-hF-1upL8iGw@mail.gmail.com>
<CA+wnhO3kT7mxbadbteppSZVcs3FRPvGL5xTY2DbrWSGohmhyVw@mail.gmail.com>
<6785e9e8-4c64-41b0-9e72-c13d1822b599@gmail.com>
<CA+wnhO0VdTP9OVdZP9bmkavma2WkE-pYpP1yW8yDwRrJ97j-2g@mail.gmail.com>
<0c3b0e1b-55d1-42b6-8c3c-4392479187db@gmail.com>
<CA+wnhO2sHcUOfi6_gvaoPMk=2DsKUOWkDjdUtGxJOguQOXDvOQ@mail.gmail.com>
<cc488d1c-6477-4e20-8bd0-cf152a2e59bf@gmail.com>
<CA+wnhO3t7xqCiDmGduxgg1Mv5M5R-GeQd0kzVcJKkUFkj91Wbw@mail.gmail.com>
<CA+hUKGLYx2GGd8BggND5QnyyJKbf1r+Q0cEYWrN+miXzaxwGNA@mail.gmail.com>
<9! 6a328e9-4fbf-433b-a8d7-071da18e12c4@gmail.com>
<3581840.1728887169@sss.pgh.pa.us>
On 10/14/24 13:26, Tom Lane wrote:
> Andrei Lepikhov <lepihov@gmail.com> writes:
>> My explanation (correct if I'm wrong):
>> OUTER JOINs allow NULLs to be in a hash table. At the same time, a hash
>> value for NULL is 0, and it goes to the batch==0.
>> If batch number 0 gets overfilled, the
>> ExecParallelHashIncreaseNumBatches routine attempts to increase the
>> number of batches - but nothing happens. The initial batch is still too
>> big, and the number of batches doubles up to the limit.
>
> Interesting point. If memory serves (I'm too tired to actually look)
> the planner considers the statistical most-common-value when
> estimating whether an unsplittable hash bucket is likely to be too
> big. It does *not* think about null values ... but it ought to.
As I see it, it is just an oversight in the resizing logic: batch 0
doesn't change the estimated_size value at all - I think because it
doesn't matter for this batch - it can't be treated as exhausted by
definition. Because of that, parallel HashJoin doesn't detect extreme
skew, caused by duplicates in this batch. NULLS is just our luck - they
correspond to hash value 0 and fall into this batch.
See the attachment for a sketch of the solution.
>
> However, this does not explain why PHJ would be more subject to
> the problem than non-parallel HJ.
Good question! I rarely touch this part of the code and maybe don't see
whole picture. But as I see it, HJ is designed differently:
repartitioning machinery is based on overall hash table size and number
of tuples and has nothing similar to 'batch 0' or parallel batches. Hash
table size is calculated for each batch and can't cause this bug.
BTW, Can we also resolve here the long-living corner case with "invalid
DSA memory alloc request size" [1]? Just because we have clear
reproduction ...
[1]
https://www.postgresql.org/message-id/7d763a6d-fad7-49b6-beb0-86f99ce4a6eb%40postgrespro.ru
--
regards, Andrei Lepikhov
Attachments:0001-Consider-extreme-skew-in-batch-0-during-Parallel-Has.patch (text/x-patch, 1.4K) download
From 87b823821abd4ef53214ba5954d8e8069a1c40fc Mon Sep 17 00:00:00 2001
From: "Andrei V. Lepikhov" <lepihov@gmail.com>
Date: Mon, 14 Oct 2024 15:48:01 +0700
Subject: [PATCH] Consider extreme skew in batch 0 during Parallel Hash Join
execution.
Batch 0 doesn't maintain the estimated_size value. In this case, extreme skew
caused by massive duplicates should be detected by the same number of tuples
left in this batch after the resizing.
This bug doesn't exist in non-parallel HJ because this batch repartitioning
procedure is based on the hash table size and number of tuples in the batch.
---
src/backend/executor/nodeHash.c | 9 +++++++--
1 file changed, 7 insertions(+), 2 deletions(-)
diff --git a/src/backend/executor/nodeHash.c b/src/backend/executor/nodeHash.cindex 570a90ebe1..319006d69a 100644--- a/src/backend/executor/nodeHash.c+++ b/src/backend/executor/nodeHash.c@@ -1242,11 +1242,16 @@ ExecParallelHashIncreaseNumBatches(HashJoinTable hashtable)
if (batch->space_exhausted ||
batch->estimated_size > pstate->space_allowed)
+ space_exhausted = true;++ /*+ * Batch 0 doesn't maintain estimated_size, so it should be+ * treated as a special case.+ */+ if (space_exhausted || i == 0)
{
int parent;
- space_exhausted = true;-
/*
* Did this batch receive ALL of the tuples from its
* parent batch? That would indicate that further
--
2.39.5
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-bugs@postgresql.org
Cc: lepihov@gmail.com, tgl@sss.pgh.pa.us, thomas.munro@gmail.com, craig@milhiser.com, pgsql-bugs@lists.postgresql.org
Subject: Re: Reference to - BUG #18349: ERROR: invalid DSA memory alloc request size 1811939328, CONTEXT: parallel worker
In-Reply-To: <3a1ac6f1-ac30-4ed4-8d0d-beb8a5aca7e5@gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox