Message-ID: <5a1bf74c-99f7-4e21-9aa2-7ae0dfe1dde0@Data-Bene.io>
Date: Fri, 13 Dec 2024 15:23:02 +0100
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: Crash: invalid DSA memory alloc request
To: Matthias van de Meent <boekewurm+postgres@gmail.com>,
 Andreas 'ads' Scherbaum <ads@pgug.de>
Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
References: <80a12d59-0d5e-4c54-866c-e69cd6536471@pgug.de>
 <CAEze2WhO235HKnc=qCh7GmjXBpL9LDPC6qQk4pvq57cZiDC7Pg@mail.gmail.com>
Content-Language: en-US
From: =?UTF-8?Q?C=C3=A9dric_Villemain?= <Cedric.Villemain@Data-Bene.io>
Organization: Data Bene
In-Reply-To: <CAEze2WhO235HKnc=qCh7GmjXBpL9LDPC6qQk4pvq57cZiDC7Pg@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Archived-At: <https://www.postgresql.org/message-id/5a1bf74c-99f7-4e21-9aa2-7ae0dfe1dde0%40Data-Bene.io>
Precedence: bulk


On 12/12/2024 22:49, Matthias van de Meent wrote:
> On Thu, 12 Dec 2024 at 22:28, Andreas 'ads' Scherbaum <ads@pgug.de> wrote:
>>
>> Hello,
>>
>> I'm running a couple of large tests, and in this particular test I have
>> a few million tables more.
>>
>> At some point it fails, and I gathered the following trace:
>>
>>
>> 2024-12-12 22:22:55.307 CET [1496210] ERROR: invalid DSA memory alloc
>> request size 1073741824
>> 2024-12-12 22:22:55.307 CET [1496210] BACKTRACE:
>>           postgres: ads tabletest [local] CREATE TABLE(+0x15e570)
>> [0x6309c379c570]
>>           postgres: ads tabletest [local] CREATE
>> TABLE(dshash_find_or_insert+0x1a4) [0x6309c39882d4]
>>           postgres: ads tabletest [local] CREATE
>> TABLE(pgstat_get_entry_ref+0x440) [0x6309c3b0a530]
> It looks like the dshash table used in the pgstats system uses
> resize(), which only specifies DSA_ALLOC_ZERO, not DSA_ALLOC_HUGE,
> causing issues when the table grows larger than 1 GB.
>
> I expect that error to disappear when you replace the
> dsa_allocate0(...) call in dshash.c's resize function with
> dsa_allocate_extended(..., DSA_ALLOC_HUGE | DSA_ALLOC_ZERO) as
> attached, but haven't tested it due to a lack of database with
> millions of relations.

IIUC the table is doubled in size when filled over 75%, so we went from 
500MB to 1GB here, doubling the number of available buckets.
It's probably good up to a point but the size limit is exceed here only 
by 1 byte and 1GB-1 are hopefully more than enough pointers.
Is it interesting to revisit the logic to increase size less quickly 
(over 500MB) ? (if at all possible given how buckets and partitions are 
managed).

There is this comment in 8c0d7bafad3 which introduce this "dshash":

There is a wide range of potential users for such a hash table, though 
it's very likely the interface will need to evolve as we come to 
understand the needs of different kinds of users. E.g support for 
iterators and incremental resizing is planned for later commits and the 
details of the callback signatures are likely to change.

I'm unsure iterators and incremental resizing has made it ?

---
Cédric Villemain +33 6 20 30 22 52
https://www.Data-Bene.io
PostgreSQL Support, Expertise, Training, R&D