Message-ID: <3d36f787-8ba5-4aa2-abb1-7ade129a7b0e@vondra.me>
Date: Fri, 4 Apr 2025 04:58:10 +0200
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: pgsql: Improve accounting for memory used by shared hash tables
From: Tomas Vondra <tomas@vondra.me>
To: David Rowley <dgrowleyml@gmail.com>,
 Tomas Vondra <tomas.vondra@postgresql.org>
Cc: pgsql-committers@lists.postgresql.org
References: <E1tzzpV-002JTf-1H@gemulon.postgresql.org>
 <CAApHDvriCiNkm=v521AP6PKPfyWkJ++jqZ9eqX4cXnhxLv8w-A@mail.gmail.com>
 <db61d8c1-7a27-4576-be37-740632a7e020@vondra.me>
 <46c20602-41e3-47b7-aa1d-7d1d2478e4f6@vondra.me>
Content-Language: en-US
In-Reply-To: <46c20602-41e3-47b7-aa1d-7d1d2478e4f6@vondra.me>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Archived-At: 
 <https://www.postgresql.org/message-id/3d36f787-8ba5-4aa2-abb1-7ade129a7b0e%40vondra.me>
Precedence: bulk

On 4/4/25 02:41, Tomas Vondra wrote:
> On 4/4/25 01:43, Tomas Vondra wrote:
>> On 4/4/25 00:57, David Rowley wrote:
>>> On Thu, 3 Apr 2025 at 04:16, Tomas Vondra <tomas.vondra@postgresql.org> wrote:
>>>> Improve accounting for memory used by shared hash tables
>>>
>>> I've not looked into why, but this is causing an issue in the
>>> join_rel_hash during add_join_rel(). See the attached script.
>>>
>>> ERROR:  pfree called with invalid pointer 0x60a15edc44e0 (header
>>> 0x0000002000000008)
>>>
>>
>> Thanks for the report and reproducer. I'll take a look tomorrow.
>>
> 
> I took a quick look, and I think the reason is fairly simple - the
> commit allocates the header and the directory as a single chunk. And for
> shared hash tables that's fine, because those have non-expandable
> directory. But the patch does the same thing for non-shared hash tables
> (not intentionally), which means that if we end up expanding the hash,
> it fails in dir_realloc(). Because hashp->dir is not a separately
> allocated chunk.
> 
> This is clearly a bug in the patch, I should have caught this during a
> review. But I'm also quite surprised none of the regression tests seems
> to expand the hash table ...
> 
> I'll think about a way to fix this tomorrow.
> 

I ended up reverting this. Unfortunately, the patch assumed the
directory is pre-allocated and not expanding in more places. I wasn't
sure how long would it take me to fix this, or how invasive the fix
would be. It seems more appropriate to revert and then maybe apply a
reworked patch (not going to happen for PG18).

Thanks for the report, sorry for missing the issue in the first place.

-- 
Tomas Vondra