MIME-Version: 1.0
Date: Tue, 25 Feb 2025 22:44:33 +0300
From: Ekaterina Sokolova <e.sokolova@postgrespro.ru>
To: pgsql-hackers@lists.postgresql.org
Subject: Proposal:  Limitations of palloc inside checkpointer
Message-ID: <db4534f83a22a29ab5ee2566ad86ca92@postgrespro.ru>
Organization: Postgres Professional
Content-Type: text/plain; charset=US-ASCII;
 format=flowed
Content-Transfer-Encoding: 7bit
Archived-At: <https://www.postgresql.org/message-id/db4534f83a22a29ab5ee2566ad86ca92%40postgrespro.ru>
Precedence: bulk

Hi, hackers!

Historically, the checkpointer process use palloc() into 
AbsorbSyncRequests() function. Therefore, the checkpointer does not 
expect to receive a request larger than 1 GB.

We encountered a case where the database went into recovery state, after 
applying all wal, the checkpointer process generated an "invalid memory 
alloc request size" error and entered a loop. But it is quite acceptable 
for the recovery state to receive such a large allocation request.

A simple solution to this problem is to use palloc_extended() instead of 
palloc(). But is it safe to allow the checkpointer to allocate so much 
memory at once?

I have proposal to update this memory allocation but I need your ideas 
and advices on how to do it in appropriate way. As an idea, we can 
replace the array with a list of arrays to allocate memory in chunks. As 
a bad idea, we can process a temporary array without locking.

I would be glad to hear your ideas and suggestions about this topic. 
Have a nice day!

-- 
Ekaterina Sokolova
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company