From: Duncan Sands <duncan.sands@deepbluecap.com>
To: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
To: Amit Kapila <amit.kapila16@gmail.com>
Cc: pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Cc: Sawada Masahiko <sawada.mshk@gmail.com>
Subject: Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
Date: Sat, 24 May 2025 17:42:30 +0200
Message-ID: <6bc28291-b212-4a84-925a-e6e5ef2fb72c@deepbluecap.com> (raw)
In-Reply-To: <OSCPR01MB14966139B6C89C27647712A90F59EA@OSCPR01MB14966.jpnprd01.prod.outlook.com>
References: <680bdaf6-f7d1-4536-b580-05c2760c67c6@deepbluecap.com>
<CAA4eK1JwJw6JOnfDxtGtSRF7kM0LbEVPRmNxWeJa5+wyoG05Xg@mail.gmail.com>
<CAA4eK1LMgqeT_bPZ3MH-VKvwOqpZyfJmF7knZhu1rqt2Pqsnwg@mail.gmail.com>
<OSCPR01MB14966139B6C89C27647712A90F59EA@OSCPR01MB14966.jpnprd01.prod.outlook.com>
Dear Hayato Kuroda, thank you so much for working on this problem. Your patch
PG17-0001-Avoid-distributing-invalidation-messages-several-tim.patch solves the
issue for me. Without it I get an invalid memory alloc request error within
about twenty minutes. With your patch, 24 hours have passed with no errors.
Best wishes, Duncan.
On 21/05/2025 13:48, Hayato Kuroda (Fujitsu) wrote:
> Dear hackers,
>
>> I think the problem here is that when we are distributing
>> invalidations to a concurrent transaction, in addition to queuing the
>> invalidations as a change, we also copy the distributed invalidations
>> along with the original transaction's invalidations via repalloc in
>> ReorderBufferAddInvalidations. So, when there are many in-progress
>> transactions, each would try to copy all its accumulated invalidations
>> to the remaining in-progress transactions. This could lead to such an
>> increase in allocation request size. However, after queuing the
>> change, we don't need to copy it along with the original transaction's
>> invalidations. This is because the copy is only required when we don't
>> process any changes in cases like ReorderBufferForget(). I have
>> analyzed all such cases, and my analysis is as follows:
>
> Based on the analysis, I created a PoC which avoids the repalloc().
> Invalidation messages distributed by SnapBuildDistributeSnapshotAndInval() are
> skipped to add in the list, just queued - repalloc can be skipped. Also, the function
> distributes messages only in the list, so received messages won't be sent again.
>
> Now a patch for PG17 is created for testing purpose. Duncan, can you apply this and
> confirms whether the issue can be solved?
>
> Best regards,
> Hayato Kuroda
> FUJITSU LIMITED
>
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-bugs@postgresql.org
Cc: duncan.sands@deepbluecap.com, kuroda.hayato@fujitsu.com, amit.kapila16@gmail.com, pgsql-bugs@lists.postgresql.org, sawada.mshk@gmail.com
Subject: Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
In-Reply-To: <6bc28291-b212-4a84-925a-e6e5ef2fb72c@deepbluecap.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox