Message-ID: <6bc28291-b212-4a84-925a-e6e5ef2fb72c@deepbluecap.com>
Date: Sat, 24 May 2025 17:42:30 +0200
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: Logical replication 'invalid memory alloc request size
 1585837200' after upgrading to 17.5
To: "Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>,
 'Amit Kapila' <amit.kapila16@gmail.com>
Cc: "pgsql-bugs@lists.postgresql.org" <pgsql-bugs@lists.postgresql.org>,
 Sawada Masahiko <sawada.mshk@gmail.com>
References: <680bdaf6-f7d1-4536-b580-05c2760c67c6@deepbluecap.com>
 <CAA4eK1JwJw6JOnfDxtGtSRF7kM0LbEVPRmNxWeJa5+wyoG05Xg@mail.gmail.com>
 <CAA4eK1LMgqeT_bPZ3MH-VKvwOqpZyfJmF7knZhu1rqt2Pqsnwg@mail.gmail.com>
 <OSCPR01MB14966139B6C89C27647712A90F59EA@OSCPR01MB14966.jpnprd01.prod.outlook.com>
Content-Language: en-US
From: Duncan Sands <duncan.sands@deepbluecap.com>
In-Reply-To: 
 <OSCPR01MB14966139B6C89C27647712A90F59EA@OSCPR01MB14966.jpnprd01.prod.outlook.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Archived-At: 
 <https://www.postgresql.org/message-id/6bc28291-b212-4a84-925a-e6e5ef2fb72c%40deepbluecap.com>
Precedence: bulk

Dear Hayato Kuroda, thank you so much for working on this problem.  Your patch 
PG17-0001-Avoid-distributing-invalidation-messages-several-tim.patch solves the 
issue for me.  Without it I get an invalid memory alloc request error within 
about twenty minutes.  With your patch, 24 hours have passed with no errors.

Best wishes, Duncan.

On 21/05/2025 13:48, Hayato Kuroda (Fujitsu) wrote:
> Dear hackers,
> 
>> I think the problem here is that when we are distributing
>> invalidations to a concurrent transaction, in addition to queuing the
>> invalidations as a change, we also copy the distributed invalidations
>> along with the original transaction's invalidations via repalloc in
>> ReorderBufferAddInvalidations. So, when there are many in-progress
>> transactions, each would try to copy all its accumulated invalidations
>> to the remaining in-progress transactions. This could lead to such an
>> increase in allocation request size. However, after queuing the
>> change, we don't need to copy it along with the original transaction's
>> invalidations. This is because the copy is only required when we don't
>> process any changes in cases like ReorderBufferForget(). I have
>> analyzed all such cases, and my analysis is as follows:
> 
> Based on the analysis, I created a PoC which avoids the repalloc().
> Invalidation messages distributed by SnapBuildDistributeSnapshotAndInval() are
> skipped to add in the list, just queued - repalloc can be skipped. Also, the function
> distributes messages only in the list, so received messages won't be sent again.
> 
> Now a patch for PG17 is created for testing purpose. Duncan, can you apply this and
> confirms whether the issue can be solved?
> 
> Best regards,
> Hayato Kuroda
> FUJITSU LIMITED
>